Imagine arguing your case in front of a judge, only to discover that several of the precedents your attorney cited simply do not exist. The cases are fictional — invented wholesale by an AI chatbot that presented them with complete, authoritative confidence. This is not a hypothetical. It has already happened, and it is happening with alarming frequency.
As lawyers increasingly turn to AI tools like ChatGPT to speed up legal research and drafting, a phenomenon known as “AI hallucination” — where large language models generate plausible-sounding but entirely fabricated information — is colliding head-on with a legal system built on verifiable facts and real precedent.
The Case That Put the Problem on the Map
The most prominent early example came in 2023, when a New York attorney named Steven Schwartz submitted a legal brief in a personal injury case against Avianca Airlines that cited at least six cases that did not exist. Schwartz had used ChatGPT to assist with research, and the AI confidently produced case names, docket numbers, and even quoted “opinions” from judges — all fabricated. When opposing counsel could not locate the cases, the court demanded explanations.
Judge P. Kevin Castel of the Southern District of New York was unsparing in his response, calling the submission “replete with citations to non-existent cases.” Schwartz and his colleagues were ultimately sanctioned $5,000. The incident made national headlines, but legal observers warned it was far from an isolated case.
Since then, similar incidents have surfaced across the country. A Colorado attorney faced sanctions after submitting a brief containing AI-generated fake citations. Courts in Texas, California, and elsewhere have issued standing orders requiring attorneys to disclose whether AI tools were used in preparing filings — a direct response to a pattern that judges can no longer ignore.
Why AI Hallucinations Are Especially Dangerous in Legal Contexts
To understand why this matters so much, it helps to understand why legal citations exist in the first place. The American legal system operates on the doctrine of precedent — courts are expected to follow the reasoning of prior decisions. Every citation in a legal brief is a verifiable link in a chain of logic that opposing counsel, judges, and clerks can examine and challenge.
When AI invents a case, it does not just make a factual error. It corrupts that chain entirely. A judge who assumes a cited case is real may be influenced by legal reasoning that was never actually written by any court. A party on the other side of the lawsuit may spend billable hours tracking down cases that do not exist. The integrity of the proceeding itself is compromised.
The underlying technical problem is stubbornly difficult to solve. Large language models like ChatGPT are trained to predict plausible text, not to verify facts against a live database. Legal citations — with their specific formats, party names, and volume numbers — are precisely the kind of structured, authoritative-sounding text these models are very good at producing and very bad at getting right.
Who Is Responsible When AI Gets It Wrong?
The legal profession’s response has been swift but uneven. Bar associations in multiple states have issued ethics guidance reminding attorneys that professional responsibility rules do not have an AI exception. An attorney who submits a false or misleading document to a court is accountable for that submission, regardless of what tool generated it.
The American Bar Association’s Formal Opinion 512, issued in 2024, addressed AI use directly, concluding that lawyers must maintain competence when using AI tools — which means verifying outputs before submitting them. In plain terms: if ChatGPT writes your brief, you still have to read it.
Some courts have gone further, requiring attorneys to certify under penalty of sanction that any AI-generated content has been independently verified. Legal research platforms like Westlaw and LexisNexis have meanwhile rushed to market AI tools specifically designed with citation verification built in — an acknowledgment that general-purpose chatbots are simply not safe for legal research without guardrails.
The deeper issue, however, is cultural. Legal practice is under pressure to move faster and cost less, and AI tools offer a seductive shortcut. Until the profession fully internalizes that AI confidence is not the same as AI accuracy, courtrooms will continue to see ghosts — cases that were never decided, judges who never wrote a word, and justice built on nothing at all.