You have driven home and arrived with no memory of the drive. Twenty minutes of lane changes, merges, pedestrians, a left turn against traffic, and your conscious mind was somewhere else entirely, rehearsing an argument or replaying a meeting. You have also walked into a room and stood there, stripped of the reason you came. Both moments are unnerving for the same reason: they briefly expose machinery that normally runs without your conscious awareness. The drive reveals how much of your competence executes below awareness. The doorway reveals what happens when the background scheduler that holds your intentions drops one.
Here is the uncomfortable translation. That doorway moment, the flicker of “wait, what was I doing,” is the permanent operating condition of an AI agent. Except the agent doesn’t get the flicker of confusion. There is no system to notice the dropped intention.
Most conversations about agent reliability are conversations about intelligence. Is the model smart enough, does it reason well enough, when does the next version ship. I want to argue that this is mostly the wrong axis, and that the industry has the cognitive framing backward.
The deeper problem is not that agents lack intelligence. It is that they lack the background machinery that makes intelligence usable: habit, prospective memory, interruption, accumulation. Once you see that, the design consequences fall out on their own, including an answer to why deterministic controls, the unglamorous hardcoded parts of an agent system, are not an admission that the model failed. They are the part of the organism we haven’t finished building.
Habits not about intelligence
The standard mental model, borrowed from Kahneman and repeated in every second AI essay: LLMs are System 1, fast intuitive pattern-matching, and the reasoning-model push is bolting on System 2 so the machine can finally think before it speaks. As a description of a single forward pass, fine. As a description of an agent doing work, it’s backwards.
Watch where an agent’s choices actually come from. In the architectures people deploy, every meaningful choice that has not been extracted into deterministic machinery routes back through the model: what to inspect next, whether a prior constraint still applies, whether to act. The agent deciding whether to check a budget cap runs the same machinery, at the same cost, with the same latitude for creativity, as the agent composing a novel strategy from scratch. There is no cheap mode. There is no reflex. Unless a builder has moved a repeated step into infrastructure, the fortieth execution is not more automatic than the first. It is another act of model-mediated judgment, with all the latency, cost, and variance that implies.
Humans are built the other way around. We are deliberation machines with an aggressive conversion pathway attached. The first time you drove a car, it took your full attention. You paid attention to everything in the mirror, were startled by every signal, sound, and movement from other drivers. The thousandth time, it costs so little that your conscious mind can vacate the premises entirely and you arrive home with no memory of the trip.
Neuroscience gives us a rough sketch of the handoff. Early practice leans heavily on deliberate control, and with repetition, more of the task shifts to procedural systems around the striatum and basal ganglia. The exact mapping isn’t important, but directionally it points to a basic fact. What began as conscious control became automatic execution, and your mental machinery was freed to work on anything else.
This conversion pathway is arguably the central trick of human competence. Almost nothing you are good at runs on deliberation. Typing, walking, reading, the syntax of your native language, the thousand micro-judgments of your profession. A professional, by default, is defined by the quality and effortlessness of these automatic judgements.
Agents don’t have this. There is no conversion pathway. Within a session, the fortieth execution of a task costs what the first one did, in tokens, in latency, and in opportunities to go wrong. Across sessions it’s worse than expensive; it’s amnesiac. The agent that performed a workflow flawlessly yesterday can’t guarantee it’ll do the same thing today. It wakes up with instructions and re-derives the workflow at the full cognitive price. Whether it succeeds or fails is a complete roll of the dice.
You can call this the deliberation tax. Agents pay retail for every decision, forever.
Habits require improvement
The obvious complaint about paying retail is the bill. Token costs, latency, the compute spent re-deriving procedures that should be free. But the deeper problem is variance.
A habit is not just cheap. A habit is consistent. When procedural systems execute your morning routine, they execute the same routine they executed yesterday. That’s the whole point. Deliberation, by contrast, is generative. It considers options. It can arrive at the answer slightly differently each time, and mostly that flexibility is the feature you wanted.
Except when it isn’t. When an agent re-deliberates a procedure it has performed a hundred times, your chance of errors are exactly the same each time; it doesn’t decrease with frequency. It can take a creative shortcut through a step that permits no creativity. It can reinterpret an instruction it interpreted correctly the previous forty times. In human terms, imagine a surgeon whose ten-thousandth suture still has to be reasoned through from the textbook. She may be brilliant. She may usually get it right. But you would not call that experience, because nothing has become automatic, and nothing is guaranteed to go the same way twice.
This is the mechanism underneath supervision decay: the tendency of agent oversight to work well in the demo and degrade in production. The decay isn’t mysterious and it isn’t about model quality. It’s what you’d predict from a stateless architecture. Humans have state, their skill improves with repetition. Agents are frozen in time, their execution repeats, and every repetition is a fresh draw from the same distribution.
The second missing system
Procedural memory is the first background system agents lack. The second is subtler and does more damage: prospective memory, the faculty of remembering to remember.
When you form an intention for the future, “pick up milk on the way home,” “circle back to that email after the meeting,” you are not holding it in conscious attention. That would be paralyzing. You subconsciously hand it to a background scheduler that watches for the trigger condition and interrupts you when it fires. The system is imperfect (the doorway moment is what its failures feel like), but it exists, it mostly works, and you generally can’t function without it.
An agent has no scheduler. What it has is a context window, and those are not the same thing. Think of it as a notepad. An instruction given at the start of a long task (“always confirm before spending,” “never touch the production config”) does not sit in a queue waiting for its trigger. It sits in a sequence of tokens, competing for salience with everything that arrived after it, and salience decays with distance. By token sixty thousand, the instruction hasn’t been rejected or forgotten in any deliberate sense. It has simply lost the competition. The intention was never held; it was written down in a place that gets progressively harder to see.
This also explains the failure mode everyone who runs agents recognizes: the agent that burns forty tool calls polishing a subtask nobody cared about while the actual objective is ignored. People describe this as the agent getting distracted, which is almost right. More precisely, it’s what deliberation looks like when it’s unsupervised. Humans have an interrupt system, an executive layer that periodically asks whether the current activity still serves the goal. The agent’s only executive layer is more of the same deliberation that wandered off in the first place. It’s turtles all the way down, and every turtle is System 2.
Put the two missing systems together and the operational picture snaps into focus. An agent is not a distractible employee. It is a mind assembled without the background machinery that makes minds functional. No habit formation, no intention scheduler, no interrupt controller, no knowledge accumulation. Pure foreground, all the way through. And once you see that, the question stops being “how do we make the model pay attention” and becomes “who is doing the background work instead?”
The simple answer is: you are. The human checking in on the agent, re-raising the instruction from an hour ago, noticing the rabbit hole, confirming the routine step for the fortieth time, is manually performing the agent’s prospective memory, its interrupt system, and its consistency guarantee. This is the supervision tax in mechanistic terms. The work wasn’t offloaded when you deployed the agent. The deliberation moved to the machine and the executive function moved to you, which is a strange trade, because executive function is what’s in short supply.
Harnesses are missing anatomy
Builders tend to experience deterministic controls as defeat. Every hardcoded check, every validation gate, every rule that fires without asking the model reads as an admission. We couldn’t trust the intelligence, so we caged it. The model is always framed as the interesting part; the harness is embarrassing plumbing that is hidden where nobody can see it.
This has the valence backward. You do not deliberate about how to walk. You would be worse at walking if you did; deliberate attention actively degrades automated skills, which is why thinking about your golf swing ruins it. Automatism isn’t the absence of intelligence. It’s where finished intelligence goes to become dependable. Your basal ganglia are not a cage around your prefrontal cortex. They are the reason your prefrontal cortex is free to do anything interesting.
The harness is the agent’s basal ganglia. Less metaphorically, the harness is where repeated deliberations get converted into cheap, consistent execution.
Applying it directly, a validation gate or a CI check is a proceduralized skill: the judgment “does this output meet the bar” was made deliberately once, by a person, and then converted into automatic execution, which is precisely the conversion the agent cannot perform on itself. A scheduled background loop is prospective memory: if an intention matters (“check pacing every morning,” “flag anomalies when they cross this line”), it does not belong in the context window, where it will decay like everything else in the document; it belongs in a scheduler, where it is structurally incapable of losing a salience competition. Storing intentions in context and hoping is the agent-architecture equivalent of trying to remember the milk by concentrating harder.
And deterministic detection over model inference, for signals that admit determinism, is reflex versus deliberation. When a metric crossing a threshold can be detected by comparing two numbers, asking a language model to notice it is spending your most expensive, most variable faculty on a job that wanted a knee-jerk reaction.
I’ve built agents around exactly this principle. The trust-bearing signals come from deterministic reads of platform APIs, and the model’s deliberation is reserved for the questions that are genuinely open, like what to do about what the reflexes found. Not because the model couldn’t do the detection. Because detection done by deliberation is detection with an error surface, and reflexes are supposed to be boring.
The design rule that falls out is almost embarrassingly simple: anything the agent does more than once, identically, is a candidate for extraction into the harness. You are performing, by hand, the conversion pathway evolution performed for you. The agent’s deliberation, the genuinely scarce and genuinely powerful thing, gets reserved for decisions that are actually novel.
| Human analogue | What it provides | Harness equivalent |
|---|---|---|
| Procedural memory (basal ganglia) | Cheap, consistent execution of judgments made once | Validation gates, CI checks, hardcoded rules |
| Prospective memory | Intentions that fire on triggers, not attention | Schedulers, commitment ledgers with due dates |
| Goal monitoring | ”Does this still serve the goal?” | Policy layers: budgets, quiet hours, interrupt limits |
| Knowledge accumulation | Experience that persists and compounds | Governed, versioned memory a human can inspect and revoke |
What the anatomy looks like in practice
A chat interface, by itself, offloads very little. It still waits for a human to notice that something needs attention, formulate the question, remember the prior constraint, interpret the answer, and decide when to check again. That is not an agent. It is a very capable instrument panel. The useful agent begins where the chat turn ends: in background loops that wake without being asked, inspect what changed, suppress what repeats, and come to a person only when judgment or authority is actually required.
The agent I have been building (and have built before) is organized this way. One of its main surfaces is Slack, which frankly doesn’t matter. What matters is that the agent decides when it is worth entering Slack at all, and every piece of that decision lives in the harness.
A scheduler decides when it wakes, so nobody has to remember to check. A ledger decides whether it has already spoken, so it never repeats itself into a muted channel. A policy layer decides whether it may interrupt, because how often a system gets to break someone’s focus is a human judgment, and not a thing the model reconsiders per message and gets differently wrong on Tuesday than on Thursday.
Promises are stored as commitments, not held as intentions: “check the pacing again in five minutes” becomes a record with a due date, and the follow-up arrives on schedule. And what the business has decided, the goals, constraints, and standing preferences, lives in a governed memory that a human can inspect, correct, or revoke, where a reversed decision supersedes the original rather than silently replacing it. A decision that quietly ages out of memory is a policy change nobody made, and human memory rewrites itself constantly and calls the result recollection. A business system does not get to lie, nor should you let it.
The model still matters. It explains, interprets, recommends, answers whatever gets asked. But the loop decides when there is something worth bringing to a person, and the harness decides what the system remembers, promises, and refuses to repeat. None of that is exotic engineering. It is schedulers, ledgers, policies, and memory, assembled around the model in the shape of the nervous system it lacks.
Improvements that compound
Now notice the property every one of those mechanisms shares.
Each began as a deliberation. Someone noticed the agent reposting the same alert. Someone got pinged at dinner by a routine digest. Someone watched a promised follow-up silently evaporate when a session ended. In the default agent-building loop, that thinking produces a prompt tweak and a resolution to keep an eye on things. The fix is stored in soft tissue, and it decays exactly the way supervision decays, because it requires attention. In the harness loop, the same thinking produces infrastructure. Each fix is a permanent acquisition and compounds for future fixes. The quiet-hours policy that governs alert tiers also defers follow-ups; the memory scoping built for one surface is inherited by the next on day one. Reliability compounds and offloads mental labor from the human to the agent.
Ask of any reliability improvement a single question: where does it live? If it lives in a prompt, a runbook, or a human’s vigilance, the work was relocated. If it lives in a scheduler, a ledger, a policy table, or a versioned memory store that outlasts every session, the work was automated. That question is the dividing line between agent systems that get better and agent systems that get supervised forever.
More intelligence will not fix this
The standing objection writes itself, usually saying that this is all temporary. Models are improving fast; next year’s agent will hold instructions better, wander less, need less scaffolding. Just wait, and the plumbing problem dissolves.
Some of it will. Longer effective context and better instruction adherence are real and compounding. But the objection misidentifies what the gap is. A smarter model is a stronger deliberator. It reasons better about the fortieth budget check. Better reasoning is more reliable in a sense, but it’s not cheaper. It holds the instruction at token sixty thousand with better fidelity. It still holds it as a notepad, not as a scheduled intention. Capability raises the quality of each deliberation; it does not create a conversion pathway from deliberation to automaticity, because that pathway is architectural. No amount of being smarter turns reasoning into reflex. Those are different functions, and in the architectures anyone actually deploys, one of them is not on the scaling curve.
Teams betting their reliability roadmap on the next model release are making a category error. They are waiting for more deliberation to substitute for infrastructure that deliberation cannot provide at any capability level, like waiting for a brilliant employee to eventually become a calendar. The employee gets more brilliant. The meetings still get missed, because remembering meetings was never a function of brilliance.
Capability and reliability are different axes. The labs are pushing one hard and only starting to push on the other with harnesses. Teams that internalize this stop asking “when will the model be good enough to remove the controls” and start asking the better question: “what has this agent deliberated about repeatedly that we should have converted into anatomy by now?”
Finishing the animal
The industry frame says constraints limit the agent, and every deterministic control is intelligence we failed to use. The anatomical frame says something closer to the opposite, that an unharnessed agent is not a freer mind. It is an unfinished one, all cortex and no spine, condemned to consciously deliberate its own heartbeat.
There’s a test I apply to agent systems generally: does this remove work, or relocate it? An agent whose executive function lives in a human supervisor has relocated the work, and relocated the worst part, the attention. An agent whose executive function lives in the harness, in gates and schedulers and reflexes that fire without asking, has actually removed it. The difference between the two is whether someone did the unglamorous work of deciding which deliberations should never happen twice.
So the next time you hardcode a check, stop feeling like you’ve failed. You haven’t overruled the intelligence. You’ve done for it what a few hundred million years of evolution did for yours, taken something it figured out once and made it free, so the expensive machinery can go be expensive somewhere that deserves it.
The models will keep getting smarter, and that’s pretty irrelevant at this point. Habits don’t emerge from intelligence, at any scale. Somebody has to build them, and for now, that somebody is you.
Sources & receipts
- Daniel Kahneman, Thinking, Fast and Slow (2011): the System 1 / System 2 framing this essay argues is misapplied to agents.
- Ann M. Graybiel, “Habits, Rituals, and the Evaluative Brain,” Annual Review of Neuroscience (2008): the striatum and basal ganglia’s role in converting deliberate control to procedural execution.
- Gilles O. Einstein and Mark A. McDaniel’s body of work on prospective memory: remembering to remember as a distinct, imperfect faculty.
- Sian Beilock’s research on explicit monitoring and choking: deliberate attention degrading automated skills.