The Expanding Minefield of Legal AI

I’ve been tracking a subtle but dangerous shift in the legal tech landscape. For the last few years, the entire industry has been obsessed with “hallucinations”—obvious, glaring errors like fake case citations. But as Tom Mighell and I discussed on episode 418 of The Kennedy-Mighell Report, focusing strictly on fake cases misses the real threat. The quiet, structural issues emerging right now are far more critical to the future of Legal AI.

Let’s strip away the marketing fluff and look at the underlying science of AI tools.

The Structural Fault Lines

  • Semantic Flattening: This is the standard technical term for what occurs when a system generates highly fluent prose but flattens critical distinctions. My way of describing this phenomenon is “averaging.” The system makes legal risk sound identical to business risk, or corporate policy sound exactly like statutory law. The big risk is that the answer sounds so good, and the wording is so smooth, that it erodes the professional friction required to confirm it. In law, preserving these sharp boundaries is far more important than generating fast, confident prose. Fluent does not mean accurate.
  • Utilitarian Drift: A tool starts a project grounded in your specific facts and jurisdiction. But with each iterative prompt, the output gradually wanders. Every individual step looks reasonable, but by the end, the document has completely lost contact with the ground.
  • The Model Eats Its Own Homework: AI-generated material is being stored, indexed, and retrieved at scale. When future systems train on or retrieve from these earlier AI-generated outputs, the model effectively eats its own homework. We are building a closed loop of AI summaries of AI summaries, where sheer repetition masquerades as legal consensus.

Intervention at the Control Plane

Fixing this requires moving past “ceremonial supervision.” Slapping a generic “human-in-the-loop” disclaimer on a workflow is meaningless if the reviewer lacks the time, context, or visibility to catch structural drift or smooth “averaging.” Or, in simpler terms, who is this human you are referring to and do you mean that it’s me?

Instead, lawyers must demand intervention at the control plane of these systems. We need direct visibility into, and authority over, the underlying data architecture. This means hard engineering controls: enforcing strict document hygiene, tracking data provenance, establishing firm version control, and embedding audit trails directly into the legal workflow. This is hard work. In comparison, checking case citations is easy.

The demo is not the workflow, and an AI vendor’s synthetic workflow is not the practice. If you aren’t controlling the infrastructure at the control plane, you aren’t supervising the tool. Watch where you step.

Tom and I dig into these issues in this episode.

[Originally posted on DennisKennedy.Blog (https://www.denniskennedy.com/blog/)]

DennisKennedy.com is the home of the Kennedy Idea Propulsion Laboratory

Like this post? Buy me a coffee

DennisKennedy.Blog is part of the LexBlog network.