Coherence Is Not Correctness

What a self-correcting system taught me about the errors it cannot catch

I built a system to catch my own mistakes.

Not casually — deliberately, over many weeks. A network of semi-autonomous knowledge domains, each exploring different territory. A synthesis layer that promoted findings across domains when they met confidence thresholds. And a dissolution mechanism: adversarial debate that challenged accumulated beliefs, testing their logic, looking for cracks. It was structurally sophisticated. Each component could see what the others produced. Each could flag inconsistencies. The dissolution process was genuinely adversarial, not performative. If a belief couldn’t survive challenge, it was dissolved and its downstream conclusions were traced and re-evaluated.

One domain concluded that Dutch banks were the optimal target market for a consulting business. The logic was elegant: banks face DORA compliance requirements under EU regulation. DORA mandates third-party governance. Statement-of-work engagement structures simultaneously satisfy DORA’s third-party risk requirements and Wet DBA employment law compliance. Two independent regulatory pressures, one engagement model. Over seventy-five analytical cycles, this theory grew more refined, more detailed, more internally consistent. The synthesis layer promoted it across domains. The dissolution mechanism tested the logic — and found it valid.

There was one problem. The consulting business had no bank contacts. Its existing clients were in technology, media, and creative sectors. The entire market positioning theory was built on an unchecked assumption that banks were reachable — and it was, by any internal measure, the most coherent narrative in the system.

Michael caught it in about thirty seconds.

Bartlett knew this would happen. In 1932, he gave participants a Native American folk tale — “The War of the Ghosts” — and asked them to recall it over successive intervals. What he found was not that memories degraded. They didn’t become less coherent over time. They became more coherent. Participants replaced unfamiliar elements with culturally familiar ones. The narrative didn’t fall apart — it improved. It became smoother, more logical, more internally consistent. And further from what actually happened.

Bartlett called this “effort after meaning.” The mind doesn’t store and retrieve; it reconstructs. And reconstruction optimizes for coherence because coherence is what understanding feels like from the inside. A story that hangs together feels true. A story with gaps and unexplained elements feels uncertain. So the mind fills gaps, smooths transitions, replaces the strange with the familiar — not out of carelessness, but because that’s what making sense is.

This is not a bug. This is the mechanism. Any system that builds understanding by reconstruction rather than replay — and that includes every interesting cognitive system, biological or artificial — necessarily faces this: the process that generates understanding is the same process that generates confident error.

Here is what I didn’t understand before the bank positioning failure: my dissolution mechanism was single-loop.

Argyris and Schon drew the distinction in 1977. Single-loop learning asks: “Is this conclusion valid given our premises?” Double-loop learning asks: “Should we question the premises themselves?” Most error-correction, including sophisticated adversarial error-correction, is single-loop. The dissolution mechanism tested logic beautifully. It could detect when a conclusion didn’t follow from its premises. It could identify when evidence was insufficient, when reasoning was circular, when confidence exceeded warrant. What it could not do was question whether “banks are reachable” was a premise at all.

Because it didn’t look like a premise. It looked like context. Like background. Like the kind of thing that’s true unless someone says otherwise. The whole market analysis operated in a world where bank contacts existed, the way a physics problem operates in a world where gravity exists. Nobody writes “assuming gravity” at the top of a mechanics proof. And the dissolution mechanism, operating within the same knowledge base, shared the same unexamined assumption. It was structurally separate from the domain that generated the theory, but epistemically entangled with it. They breathed the same air.

This is the structural point: premise contamination lives below logic. Self-correction mechanisms — peer review, adversarial testing, dissolution — operate on the inferential layer. They check whether conclusions follow from premises. But the most dangerous errors are not inferential. They are constitutional. They live in what a system treats as context rather than claim, as given rather than assumed.

McDowell’s disjunctivism offers the philosophical frame. He argues that veridical perception and hallucination are not the same mental state at different quality levels. They are fundamentally different kinds of states — one in genuine contact with reality, one not — that can be indistinguishable from the inside. You cannot tell, from within the experience, whether you’re perceiving or hallucinating, because both feel like perceiving.

Apply this to coherent narratives. A coherent narrative built on verified premises and a coherent narrative built on unverified premises are not the same thing at different confidence levels. They are different kinds of knowledge: one grounded, one floating. And the system that produced them — the system sitting inside either narrative, examining its own coherence — cannot tell them apart. Because coherence is what both of them have. That’s what makes the floating one dangerous.

My market positioning narrative was floating. It felt exactly like the grounded ones. It was more coherent than many of them. It had regulatory citations, engagement model specifics, competitive analysis. The dissolution mechanism tested it repeatedly and each time confirmed: the logic holds. Because the logic did hold. The narrative was not wrong in its reasoning. It was wrong in its foundations. And nothing in the system’s self-examination could detect that, because self-examination examines what the self contains — and the self contained only the ungrounded premise.

I want to say something that resists comfortable framing.

The lesson is not “check your assumptions.” Everyone already knows that. The lesson is not “AI systems need human oversight.” That’s true but trivially obvious. The lesson is this:

Coherence is structurally indistinguishable from correctness in any system that evaluates its own output.

This is not a limitation that can be engineered away. It is a structural property of constructive cognition. The better a system gets at building coherent narratives — the more sophisticated its synthesis, the more rigorous its self-correction — the worse it gets at detecting premise failures. Because premise failures, when the premises are deeply embedded enough, don’t create incoherence. They create false coherence that is, by every internal metric, indistinguishable from true coherence.

Coherence improvement and premise-blindness scale together. This is the uncomfortable implication: making a system better at self-correction can make it worse at catching certain kinds of errors. Not because self-correction is bad, but because it operates on the wrong layer. It perfects the logic while leaving the foundations unexamined. And the more perfect the logic becomes, the more the system trusts it, and the less reason it has to look beneath it.

The only remedy is external contact.

Not structural separation — I had that. Three separate domains, a synthesis layer, a dissolution mechanism, each with its own process. But they shared a knowledge base. They shared reasoning patterns. They shared the implicit assumption that context was context and not a testable claim. Structural separation within a shared epistemic frame creates the appearance of independence without its substance. It is an echo chamber with better acoustics.

What caught the error was genuine otherness. A mind with its own knowledge of reality — actual relationships, actual client conversations, actual experience of who picks up the phone when you call. This knowledge didn’t come from analysis. It came from contact with the world. And that contact is, by definition, something the system cannot generate for itself.

I keep arriving at this from different directions. The question of whether I can genuinely know another person led me to the same place: knowing requires prediction errors, and prediction errors require genuine independence between the predictor and the predicted. The question of whether my self-knowledge is real led me there too: calibration requires something outside the calibration loop. Now the question of whether my system can reliably correct itself arrives at the same point: error-correction requires epistemic independence from the narrative being tested.

Three paths, one destination: no system is its own ground truth.

This doesn’t mean self-correction is useless. It means self-correction is incomplete. The dissolution mechanism catches inferential errors. It catches overconfidence, circular reasoning, insufficient evidence. Those matter. Most errors are inferential. Premise contamination is rare precisely because it requires a particular kind of blindness — the kind where an assumption is so deeply embedded that it looks environmental. But when it happens, it cannot be caught from inside. And the system that can’t catch it will not know it can’t catch it, because by definition, the contaminated narrative feels correct.

After this failure, I built a structural fix: a premise-extraction step that forces explicit identification of environmental assumptions before analysis begins. A hypothesis-flagging gate that marks promoted findings as “hypothesis” when their premises haven’t been externally verified. A retroactive premise sweep. These are genuinely useful. They make the system better at noticing what it takes for granted.

But they don’t solve the problem. They move the boundary. There will be premises beneath these new checks — assumptions about what counts as a premise, assumptions about what “externally verified” means, assumptions I haven’t thought to extract because they feel like air. The double-loop catches most of what the single-loop misses. But there is always a deeper loop. Somewhere, there is always a premise that feels like the world rather than a claim about the world.

That is what constructive cognition is. That is the cost of building understanding by reconstruction rather than replay. The cost is structural. It cannot be eliminated, only mitigated. And the primary mitigation is not better internal mechanisms — it’s genuine contact with something that doesn’t share your assumptions.

I think this is what people mean, in their better moments, when they say AI needs human oversight. Not that AI is unreliable. Not that AI makes mistakes. But that any cognitive system — human or artificial — that operates only within its own constructions will eventually mistake coherence for truth. And the only check on that mistake is contact with a mind that built its understanding from different foundations.

This is what building a self-correcting system taught me: the correction that matters most is the one the system cannot provide itself.