AI Hallucinations Aren't Just a Technology Problem

A language model that invents a case citation is doing exactly what it was built to do. The lawyer who files it under their signature is not. That distinction has been getting lost since generative tools entered legal practice in earnest, and the cost is showing up in sanctions orders and disciplinary referrals. The word that organizes most of the discussion — hallucination — points at the machine. It names a defect in the tool, a thing to be measured, benchmarked, and eventually engineered away. But the failure that reaches a judge does not happen inside the model. It happens when a person decides that plausible text is true and puts their name behind it.

Start with what a hallucination actually is. A language model produces the most probable next stretch of text given everything before it. When it generates a citation, it assembles something with the shape of a citation — a plausible party name, a plausible reporter volume, a plausible court — because that shape is what the training data says belongs there. Fluency is the entire product. The model has no representation of whether the case exists, because existence is not a category it operates in. Asking it not to hallucinate is, strictly, asking it not to do the thing it does.

The more useful place to look is not the model but the practice around it. Long before generative tools, the premise was simple: a lawyer answers for what they file. A misremembered holding, a summary an associate wrote without opening the opinion, a citation copied from a brief nobody re-checked — all of them are the same kind of mistake, and the tool that supplies the bad material has never been the thing that excuses it. A model that fabricates a citation is new only in how fluently it does it.

That fluency is also what makes the lapse easy, and it almost never looks like carelessness. A lawyer under deadline asks a model a question and gets back something specific and polished, formatted exactly like law, with case names, pin cites, a reporter volume, and the confident cadence of a brief that has already been checked. Nothing about it looks provisional. That is the trap: the output carries every surface signal that separates finished work from a first draft, and none of the work those signals usually reflect. The cases were never pulled, but the page does not say so. The one step that would catch it — reading the cases themselves — is also the step that feels most skippable, because nothing on the screen suggests it is needed.

None of this is how the problem is usually framed, which is why the proposed solutions keep missing it. The first response is to wait for the technology to fix itself, on the expectation that the next generation of models, or retrieval systems anchored to real databases, will push the hallucination rate low enough that the problem evaporates. It is a convenient belief: the vendors have every reason to promote it, and their buyers every reason to accept it, which makes it worth asking how much rests on genuine confidence in a coming fix and how much is marketing dressed as inevitability. The rate will keep falling, and that part is probably right. But accuracy approaching certainty is not certainty, and a tool wrong one time in a hundred still needs a human to find the one. You cannot know which output is the bad one without checking, and once you are checking all of them, the model's accuracy has relieved you of nothing. It has only made the failure rarer, and therefore easier to stop expecting.

A second response is to bolt verification onto the tool: citation checkers, systems that confirm each reference against an actual database before it reaches the draft. This is genuinely useful and worth building. It does not, however, change who is on the hook. A checker can do part of the lawyer's job faster and shrink the surface where a fabrication slips through, but the smaller surface is still the lawyer's to watch, and when a brief is wrong no court accepts the explanation that the verification layer failed.

A third response gives up on the tools altogether, on the theory that a ban removes the risk. But a lawyer who would file an unread case from a model is a lawyer who would file an unread case from a stale internal memo or a confidently wrong colleague. The tool did not create the verification gap; it only widened the opening and lowered the effort needed to fall through it. Banning it closes one opening and leaves the gap, and the lapse simply finds the next instrument. Every one of these fixes aims at the machine, when the thing that needs fixing sits upstream of it, in the person deciding what to sign.

What the situation requires is nothing new. A lawyer's work has always involved material that cannot be taken at face value, and the long-settled way of handling it is to check leads against their sources before relying on them. A book review is not the book, a scientific abstract is not the peer-reviewed study, a summary of a contract is not the contract. Each is a place to begin the work, not a way to skip it. What a model gives back is no different: a fast, often useful answer with the same standing as a knowledgeable colleague's confident reply, and trusted only once someone has checked it.

A related point concerns the instrument itself. A tool whose defining trait is fluent invention has to be used with that in view, and using it without grasping what it produces is its own kind of error, separate from any single fabricated citation. A lawyer need not understand how the model works, only what sort of thing it produces, and handle it accordingly. None of this calls for new rules. The expectation that whoever signs a filing is responsible for what it contains long predates the technology, and being taken in by a convincing tool has never been much of a defense. What has changed is the stakes and the frequency, not the question beneath them.

None of this is an argument against the tools, which are saving capable lawyers real time. It is, instead, an argument against the profession reaching for "the AI hallucinated" when the person using the tool is almost always at the center of what goes right or wrong with it. Relocating a human failure into a machine is comforting but inaccurate: there, it reads less like a lapse than like a glitch, something that happened to the lawyer rather than something they did. And while that framing holds, the real limits stay obscured, the lawyer's and the tool's alike, and neither can improve.

Related: The Clock Mismatch in Enterprise Legal Innovation · What a Vendor's Posture Toward Services Reveals · Without a Constraint, AI's Energy Bill May Never Shrink