By Rupali Patel Shah, Head of Legal Solutions, DiliTrust
A few weeks ago, a law firm filed a brief that cited cases that don’t exist. The cases were detailed, plausible, and entirely fabricated by an AI tool. Around the same time, a video of Sam Altman circulated where he explains, with notable candor, that AI lacks temporal intelligence — it genuinely cannot time someone running a mile, even as the chatbot confidently insists it can.
Two stories. Both widely covered. Both met with the same collective response: well, that’s embarrassing, but AI will get there eventually.
And that word — eventually — is exactly the problem.
The “…yet” we can’t stop adding
Every conversation about AI limitations has an implied ellipsis at the end. AI hallucinates facts …but that’ll improve. It can’t understand time …yet. It makes things up with complete confidence …for now.
I’ve been thinking about this a lot, and I’ve come to a conclusion that I think a lot of people quietly agree with but are reluctant to say out loud: hallucinations are not a bug to be patched. They are a structural feature of how AI works. And the sooner we accept that, the sooner we can stop being surprised — and start being smarter about what we’re actually using this technology for.
Here’s why hallucinations are permanent: AI is built to find answers. That’s not a flaw in the design; that’s the design. It is trained on the principle that a response is required and it will generate one. When the data doesn’t support a complete answer, the model doesn’t pause and say “I don’t know”. It fills the gap. It is constitutionally incapable of sitting with uncertainty the way a cautious human expert would — the kind who says “let me check that before I put my name on it”. And before anyone comes at me with “well, you need to train the AI to not make things up,” just know that statement is the whole point.
No amount of additional training eliminates that drive to answer. You can constrain it, weight it differently, add guardrails, verify and teach AI the correct answer. But all of that takes human intervention and the underlying architecture is solution-seeking. Always.
The accountability problem nobody wants to talk about
There’s a principle I find myself returning to constantly, one often loosely credited to IBM (though the actual origin is murky): AI cannot be held accountable, therefore it should not be making decisions.
Think about what accountability actually requires. It requires consequences-both good and bad — intense enough to motivate better behavior, thoughtful behavior. An attorney who cites fabricated case law faces sanctions, bar complaints, professional ruin. A doctor who misdiagnoses based on a hunch rather than evidence faces malpractice. An employee who cuts corners faces termination. Consequences shape behavior. They create the conditions under which judgment gets exercised carefully.
AI has none of that. It cannot be fired. It cannot be sanctioned. It cannot experience financial hardship or professional embarrassment. It has, as the saying goes, zero skin in the game. Its outputs are not tied to any form of continued existence or wellbeing. It is trained to produce answers — and produce them it will, even if it has to invent the supporting evidence.
Think about what happens to any system when you remove consequences from the equation. Quality deteriorates. If it doesn’t matter to you whether a court finds your argument credible, why not fill in the gaps with plausible-sounding citations? If there’s no penalty for a wrong answer, why pause at all? The Sullivan & Cromwell situation isn’t a story about a careless tool. It’s a story about a system doing exactly what it was built to do — in a context where the consequences of being wrong belong entirely to a human being who wasn’t paying close enough attention. That is an example of human failure, not a technology fail.
You can’t teach judgment, but you can teach process
I say this so often that my colleagues now finish the sentence for me. And I keep coming back to it when thinking about AI because it is, I think, the cleanest way to understand both what AI can do and what it cannot.
Judgment is the capacity to weigh competing considerations, account for context, absorb ambiguity, and arrive at a decision you are willing to stand behind — knowing the outcome is yours to own. Process is the repeatable, structured set of steps that gets you from input to output reliably. Judgment cannot be taught. Process absolutely can.
AI is genuinely extraordinary at process. Algorithm-backed, high-volume, pattern-recognition work — it is a real force multiplier when used the right way. The mistake we keep making is treating that as a limitation to overcome rather than a clearly defined capability to use well.
The right mental model for working with AI isn’t trust but verify. It’s save time creating so you can spend more time verifying. Offload the high-volume, high-effort, low-value work. Automate what is genuinely automatable. Use AI to generate the first draft, the summary, the initial scan — and then bring a human with actual skin in the game to review it.
Stop trying to give it a watch!
Here’s what frustrates me about a lot of the AI capability conversation: we keep trying to solve problems that are already solved.
Temporal intelligence — the ability to track the passage of real time — is already available. It’s called a watch. A basic one costs less than a cup of coffee. And yet there are serious people investing serious resources into trying to teach a large language model how to time a mile run. Talk about a real Rube Goldberg machine!
The broader pattern is the same. We are taking something that is powerful, computationally expensive, and genuinely useful for complex tasks — and pushing it into territory where a simple, cheap, reliable tool already does the job. Why? Because we are so dazzled by the capability that we can’t stop asking it to do things that don’t require it.
There’s a word for what happens when you rush a product into mass adoption before it’s ready, in markets and for use cases it wasn’t designed to serve. Cory Doctorow coined the term: enshittification. The technology becomes less useful, expectations get warped, and the genuine capabilities get buried under a pile of bad implementations.
We are doing this to AI in real time. And the people who will pay the price are the ones using it for consequential decisions without understanding what they’re actually holding.
Jack-Jack doesn’t know what he can do yet
There’s a scene in The Incredibles where the baby, Jack-Jack, starts spontaneously manifesting superpowers — and nobody around him quite knows what to do because he doesn’t know how to control them. He is enormously powerful, wildly unpredictable, and not remotely ready for any kind of deployment.
That’s AI right now.
The potential is real. The flashes of capability are genuinely impressive. But we don’t have the infrastructure to support the demand we’re already generating. The energy requirements alone are staggering — and we are nowhere close to a sustainable solution. We are being sold a product that is, in many ways, still learning what it is. And the business model for getting it there involves convincing — or compelling — as many people as possible to use it now, so that it can learn from their interactions and eventually become what it is already being marketed as.
That’s a strange thing to ask people to participate in, especially when the cost of the errors falls on the human and not the tool.
What this actually means for how we work with AI
None of this means AI isn’t worth using. It absolutely is — in the right contexts, with the right verification, for the right work.
We should stop waiting for AI to develop judgment it will never have and stop expecting it to hold itself accountable in ways that are structurally impossible. Stop being surprised when it hallucinates and stop treating every hallucination as evidence of a gap that will someday be closed.
The two truths that keep coming back to me are simple. You can teach process, but not judgment. And AI cannot be held accountable because it is a tool, not a person.
Take those two things seriously and the rest of the conversation gets a lot clearer. AI belongs in the process layer — surfacing, summarizing, structuring, drafting, scanning, sorting. Humans belong in the judgment layer — deciding, validating, owning the outcome, facing the consequences.
Hallucinations aren’t going away. But they don’t have to be the problem. The problem is using a tool built for process as a substitute for judgment — and then being surprised when nobody pays the price for getting it wrong.
Somebody always does. It’s just not the AI.
