South Korean courts crack down on AI hallucinations causing fake case citations

South Korea's judiciary is tackling a growing problem where AI hallucinations lead lawyers to cite fake court cases in legal documents. A judge in Seoul said their workload has increased significantly as each cited case must now be verified individually.

In one case, a lawyer in Daegu High Court cited a non-existent Supreme Court ruling, while another in Gwangju referenced Article 451 of the Civil Procedure Act with unrelated content. Some lawyers admitted using Google's Gemini large model for case searches without fact-checking.

To address this, South Korea is pushing to amend laws to fine lawyers who submit false legal references. Court procedures will also be updated to require parties to disclose AI use in drafting legal documents.

The judiciary has allocated 16.1 billion won (about $6.4 million) to build a dedicated AI platform. A pilot version of an AI trial assistance system, including Supreme Court rulings and legal manuals, launched in February, with plans to integrate generative AI.