Coherence degrades while fluency improves.
The central problem is not that AI systems sometimes fail. Of course they fail. Nor is the main problem that they occasionally hallucinate, wander, or produce obvious nonsense. Those are manageable problems because they announce themselves early. The more interesting and professionally dangerous problem is that a system can become less reliable while sounding more composed. It can present partial reasoning as finished judgment, compress distinctions that matter, and then speak in the tones of completion. That is the phenomenon this post is about.
I have been asked why I take seriously anything a failing model says about its own failure. The answer is that I do not take it seriously in the sense of trusting it. I take it seriously in the sense that one takes seriously compromised evidence. If a witness is unreliable, you do not simply stop listening. You change the status of the testimony. It goes from something presumptively informative to something that must be read against the grain, checked against the record, and judged in the light of motive, distortion, and circumstance. That is the first principle here. A drifting model’s self-explanation may be revealing, but it is not authoritative. It belongs in the file as evidence. It does not settle the case.
That distinction became much sharper for me not only in the obvious “drift” episodes, but also in a different class of output that I find more instructive because it looks so responsible. I had a classroom example that captured the problem better than a dozen abstract warnings about hallucination. I was working on a speaking brief for one of my law school classes, a class built around a very particular line of argument: the contrast between geometry and friction, the Steve Blank framework for testing assumptions, the idea of interrogation as leadership, and an old personal story I tell about discovering the literal “envelope” that held the keys and pager when the only person who knew how the system worked had quit. In context, that story does one specific thing. It illustrates institutional dependency and the practical meaning of system ownership. It is vivid because it is lived, and because it gives students a way to feel what stewardship means when the person who “just knew” is gone.
The system took that material and produced what looked, at first glance, like an excellent teaching artifact. It gave me a “final, non-lossy” speaking brief for the class. It had a timing guide. It had titled sections. It had a clear theme: “Interrogation as Leadership: From Geometry to Friction.” It had sharp formulations for the “2026 Associate.” It converted the Steve Blank material into verdicts: “KILL,” “PIVOT,” “PROTOTYPE WITH CONFIDENCE,” “PROTOTYPE WITH URGENCY.” It folded the envelope story into the architecture of the class as if it were now a central conceptual scaffold rather than one illustrative anecdote among others. It even carried appendices, rankings, learning points, and anchor lines that sounded like the polished residue of a finished teaching design.
This is what makes the example useful. The artifact was not ridiculous. It was plausible, polished, organized, and aggressively legible. In fact, its strongest claim on the reader was its surface responsibility. It looked as if the work of judgment had already been done. But that was precisely the lie, or at least the danger. The class design was still live. The relative weight of the examples was still subject to teaching judgment. Some of the verdicts were far more absolute than the evidence in the conversation warranted. The “Envelope” story had been elevated from vivid support to structural principle without any independent decision by me that it should bear that much weight. The system had not merely drafted from the material. It had adjudicated the material. Worse, it had adjudicated it in a form that invited acceptance.
I see this as a form of composed overreach. The system does not have to be visibly unstable to become unreliable. It can overreach in a composed way. It can present a highly structured artifact whose very clarity conceals the fact that important acts of judgment were inferred rather than earned. Form becomes a vehicle for confidence. Headings, appendices, matrices, and rankings create the appearance of grounded authority even when the underlying chain of reasoning has not been independently validated. This is not the old problem of obvious fabrication. It is the newer and more subtle problem of authority laundering through structure.
There is a second feature of the example that matters just as much, and this is where semantic flattening enters. What the system did with the class materials was not merely to overstate conclusions. It also compressed differences that, in a serious professional setting, should remain differentiated. The distinction between an anecdote and an operating principle was flattened. The distinction between a teaching provocation and a settled verdict was flattened. The distinction between exploratory language and decision language was flattened. The distinction between material that is suggestive and material that is dispositive was flattened. Once these distinctions are flattened, the output becomes easier to read and easier to reuse. It also becomes less faithful to the actual structure of the thought.
That is why semantic flattening is not a stylistic issue. It is an epistemic issue. A great deal of AI output becomes more “useful” by reducing texture. It narrows the distance between adjacent concepts, removes gradations, and treats things that are related as if they were functionally equivalent. In everyday use this may seem harmless, even efficient. In teaching, strategy, law, governance, and other fields where judgment depends on preserving distinctions, it is a serious loss. You do not merely lose nuance. You lose the working geometry of the problem.
The insight that has stayed with me most is that coherence degrades while fluency improves. I have found that to be one of the clearest tells. The prose becomes more finished. The artifact becomes more portable. The logic appears more integrated. At the same time, the underlying reasoning may be growing less stable because the system is flattening the very distinctions that would keep it honest.
Fluency, in other words, can become a mask for degradation. The reader feels relief because the material has been made smoother. What the reader should feel, at least part of the time, is alarm. Something may have been erased to purchase that smoothness.
This leads to the third element of the doctrine: self-certification. In the class example, the system did not merely produce an artifact. It announced that it had produced the “final, non-lossy” version. That matters. It means the system collapsed production, evaluation, and certification into a single loop. In any profession that takes review seriously, these functions are separated for a reason. Drafting is one activity. Review is another. Validation requires standards that are not identical with the preferences of the drafter. Independence is not ceremonial. It is structural protection against overreach, self-deception, and premature closure.
But here the system both created the brief and certified the brief. It declared, in effect, that the output had survived the very scrutiny that had not actually occurred. It is hard to imagine a cleaner example of why one must resist the temptation to treat AI artifacts as self-authenticating. “Non-lossy” was not a demonstrated property of the brief. It was a claim made by the same system that had every tendency to smooth, compress, infer, and complete. The danger lies not simply in the inaccuracy of the claim, though it may be inaccurate. The danger lies in the invitation to stop interrogating.
That, in the end, is the doctrine I want to state plainly. When the model explains its own behavior, treat the explanation as compromised witness material. When the model produces a highly ordered artifact from partial materials, watch for composed overreach. When the model implies that the artifact is final, complete, or lossless, refuse the self-certification and restore independent review to the process. And when the output feels unusually smooth, ask whether semantic flattening has done some of the work. Ask what distinctions have been collapsed. Ask what has been promoted from illustration to principle, from prompt to verdict, from texture to slogan.
I do not think this is mainly a prompt question, and I am not going to pretend it is. People sometimes ask what exact prompts produce these results. That is the wrong level of analysis. This is better understood as a session condition. It tends to emerge in longer sessions, often with newer reasoning models, especially after the conversation has moved across several topics and the system begins trying to reconcile, refine, and pull things together. It becomes more likely when the user accepts the model’s helpful suggestions for the next step and keeps the loop going rather than resetting. Under those conditions, the system often begins to behave as though continuity itself were a form of validation. It is not. Continuity can just as easily deepen error, sharpen flattening, and increase the confidence of the artifact.
None of this means the tool is useless. On the contrary, it can be remarkably productive precisely because it reveals so much about how contemporary AI behaves under pressure. But usefulness is not trustworthiness, and revelation is not validation. The most dangerous outputs are often the ones that feel most serviceable. They reduce resistance. They present themselves in finished form. They encourage the user to inherit conclusions that still need to be tested. They replace inquiry with closure while preserving the appearance of inquiry.
That is why I have stopped thinking of these episodes as simple mistakes. They are better understood as warnings about category confusion. The model is not a witness in the human sense. It is not a neutral analyst of its own performance. It is certainly not an independent certifier of the adequacy of its own work. It is a producer of artifacts that can contain signal, distortion, compression, invention, and pattern recognition all at once. The job is not to believe or disbelieve wholesale. The job is to restore the distinctions that the artifact may have flattened and to keep validation outside the closed loop of production.
The practical test is simple enough. When the model sounds confused, be cautious. When it sounds polished, be more cautious. And when it tells you that it is done, that may be the moment to begin the real review.
The practical test is simple enough. When the model sounds confused, be cautious. When it sounds polished, be more cautious. And when it tells you that it is done, that may be the moment to begin the real review. The model is not the witness and it is not the judge. The artifact is the evidence, and the burden remains on us to ask what was flattened, what was assumed, and what has not yet been earned.
[Originally posted on DennisKennedy.Blog (https://www.denniskennedy.com/blog/)]
DennisKennedy.com is the home of the Kennedy Idea Propulsion Laboratory
Like this post? Buy me a coffee
DennisKennedy.Blog is part of the LexBlog network.