Developed Independently: A Practitioner’s Answer to Gorelkin’s Open Question

A response to Mikhail Gorelkin’s “Category Theory as a Language for Understanding Large Language Models”

Mikhail Gorelkin’s recent piece on category theory and LLMs is one of the more honest and structurally rigorous contributions to a conversation usually conducted with much shallower context. His central argument — that we are stuck describing large language models in the wrong formalism, the way quantum mechanics seemed paradoxical when forced through set-theoretic intuition — is not rhetorical ornament. It is a real diagnosis of a real problem.

Gorelkin himself identifies the gap his framework leaves open. After building a compelling philosophical case for categorical language as the right way to think about LLMs, he acknowledges that the crucial question remains unanswered: does this framework generate new predictions and engineering solutions, or is it primarily a more elegant redescription of what we have already observed? While I agree that redescription has genuine value if it adds something new, it is not necessarily explanatory, and Gorelkin is careful enough to know it.

I want to offer an answer to that question from the ground up, not from a mathematical or philosophical perspective. I spent a lot of time since my retirement thinking about building a household AI. Starting this year, I began coding and testing a personal assistant on top of current LLM infrastructure. In that process I ran into exactly the structural limitation that Gorelkin described, diagnosed it in practical terms, and built the additional architectural layer his framework prescribes. Until I read his article, I did not realize I had in fact implemented category theory.

I think that parallel convergence itself is worth examining.

The Surplus Structure Problem in Practice

Gorelkin borrows the concept of “surplus structure” from Jonathan Bain’s work on quantum mechanics. Set-theoretic language brings with it elements and relations that do not correspond to physical reality. When we impose this surplus structure on quantum systems, we get seemingly absurd paradoxes. When we do not use the surplus structure, the apparent paradox goes away.

The practical equivalent in LLM-based systems is the assumption that a language model, given sufficient context, will naturally and reliably track what is true over time. This is the surplus structure we import from our experience of human memory and human reasoning — the expectation that what is known persists, that corrections take hold, that what has been dismissed stays dismissed.

It does not. And the failure mode is not random error. It is structural.

In the system I am building — which I call Sage — the base conversational layer extracts and stores what is semantically plausible in context. It is optimized for coherent, contextually appropriate continuation. It is not optimized for truth as a persistent invariant. The result was a memory system that accumulated aspirational language as fact, surfaced corrected information as if the correction never happened, and gradually drifted in ways that were individually small but collectively corrosive.

I spent a lot of time patching this at the surface — tightening extraction prompts, adding defensive guardrails, running cleanup scripts against accumulated noise. The patches helped at the margins. The underlying problem remained. Eventually the diagnosis became unavoidable: I was trying to solve a structural limitation by adding surface-level constraints to the wrong layer.

The Categorical Insight Arrived at from Practice

Gorelkin writes: “categorical language — which begins with morphisms and defines objects through their relations — proves more natural” than set-theoretic language when describing systems where meaning is relational and distributed rather than atomic.

Around the same time I was wrestling with Sage’s memory failures, the same structural insight emerged from a completely different direction: if you let an LLM extract entities and relationships freely, it will invent new entity types per document, create vague edge types, and duplicate the same entity under slightly different names. At ten documents this looks fine. At a thousand it is unusable. Better prompting will not fix this. You need an ontology — a defined contract that specifies what can be extracted, what types of relationships can exist, and where the meaning actually lives.

The key structural observation is that meaning and weight belong on edges — on the relationships between entities — not on the nodes themselves. An entity tagged “important” in isolation tells you nothing. An entity that holds a high-weight commitment edge to a named person, connected to a recurring pattern of behavior, connected to a recent conversation — that is where the semantic content resides.

This is, precisely, the categorical principle Gorelkin is describing. Objects derive their significance from the morphisms connecting them, not from intrinsic properties. The node is almost incidental. What matters is the structure of relationships around it.

I did not derive this from category theory. I arrived at it because a memory system that scored nodes rather than edges kept giving me wrong answers.

The Additional Architectural Layer

Gorelkin’s prescription for the hallucination and drift problem is stated clearly: “The solution, then, lies not in the naive demand to ‘teach the model to tell the truth,’ but in understanding the structural limitations and in designing additional architectural layers that make truth-tracking a separate system constraint.”

This is exactly what the ontology approach implements.

In Sage’s current architecture, the base conversational layer remains unchanged. It does what it does well — contextually coherent, semantically rich and fast. Running parallel to it, yet entirely separate, is an extraction pipeline writing commitments made, opinions held, and relationships between entities to dedicated tables. These tables have their own logic, invariants, and integrity constraints. They do not ask what is plausible in context. They ask what was explicitly stated, by whom, with what degree of confidence, and when.

The scored edges between entities carry the weight. A commitment made three times across separate conversations, consistently reinforced, connected to a named person and a specific domain — that edge carries high weight and surfaces reliably. A single aspirational statement made once in passing carries low weight and does not. The system does not need to decide in the moment whether something is true. It has a structural record of what has been established and how firmly.

Truth-tracking is not a property of the base layer. It is a separate system constraint, implemented as a separate architectural layer, with its own schema and its own retrieval logic.

The framework Gorelkin describes is not merely a more elegant description of what already exists. It is a blueprint for what has to be built — and the system I am building is one working implementation of it.

Where I Would Push Back

There is one place where I think Gorelkin’s framing deserves direct challenge.

Near the end of his piece he offers what he calls “the most provocative perspective”: that LLMs show us what human language looks like when torn from the human — from the psyche, the ego, the sensorimotor apparatus. He suggests that the human psyche may be surplus structure, an additional layer on top of language rather than its foundation. Much as classical physics is a special case of quantum mechanics, the human mind may be a special case of something more fundamental that the transformer reveals.

It is a genuinely interesting provocation although I think it inverts the actual relationship, and the inversion matters for how we build systems.

While building and using Sage, the pattern is consistent: the LLM provides breadth of information and speed of synthesis, at a scale no human can match. The human, however, provides something the architecture cannot generate internally — grounded judgment, lived context, the ability to say this is true because I have lived it and not because it is the most plausible continuation.

The truth-tracking layer I described above is not autonomous. It is calibrated against human correction, human confirmation, human judgment about what matters and what should be dismissed. The architectural layer that makes truth-tracking a system constraint is, at its root, a mechanism for anchoring the relational machinery to human experience over time.

This is not the human as surplus structure. This is the human as the foundation on which the system’s accuracy depends. The LLM is the additional structure — the amplifier — not the other way around.

Gorelkin is right that the relational structure of language operates without a psyche, but a system drifts when it operates only on relational structure without grounding in human judgment. The evidence is in the memory logs.

The Conversation That Still Needs to Happen

None of this diminishes what Gorelkin has stated. The categorical framing is genuinely clarifying, and the prescription — additional architectural layers for truth-tracking, structural rather than surface-level solutions — is correct. The work I have described here is early and ongoing. Sage’s ontology layer is weeks old and still in observation. The answers about whether the edge-weighted relational model produces the right retrieval behavior at scale are not yet in.

However, the direction is validated, and it came from the same structural insight that categorical language describes — arrived at independently, from practice, without the formalism.

That, perhaps, is the real answer to Gorelkin’s open question. He asked whether categorical language generates genuine engineering solutions or is primarily a more elegant redescription of what we already observe. The answer, arrived at from the ground up and without the formalism, is that it generates the right solutions. When practitioners independently build what the framework prescribes without knowing the framework exists, that is strong evidence that it is describing something real.

The author is developing Sage, a personal household AI assistant, and writing a series on human-AI collaboration called “Rethinking the Human in AI.”