Yegge's Counting House: Autonomy Earned Per Intent, Recorded in a Policy Table

“CH models initially have no autonomy. They earn higher levels through evals, with the results recorded in the policy table.” That sentence, from Steve Yegge’s essay The Counting House, describes a design for high-volume casework built around cheap models. Every intent starts at zero. Each one climbs only when it proves itself against cases that humans have already scored.

One artifact holds the design together, and it is small enough to copy. Read it as a table that records, for each intent, how far the model is trusted on that intent today.

Our reading is that this table is the part a customer service leader should take away. Governance, written down this way, is what can make a cheaper model usable in front of a customer queue. It moves the decision about which model to run from a vendor conversation into a document your own team owns.

The table holds the autonomy

Yegge names five tiers: off, shadow, assist, contain and transact. Promotion happens by eval. Demotion happens on a bad score. The policy table records where each intent sits.

The design choice that matters is granularity. Autonomy belongs to the intent, so a model can hold one tier on a simple question and a lower tier on a refund at the same time. A team never has to answer “do we trust the model?” in the abstract. It answers a narrower question, row by row: has this model earned this tier on this kind of case?

We have written before about autonomy scores that are earned slowly and revoked fast. The Counting House adds a concrete place for that score to live. Kept in a table, a tier could be read by an auditor, diffed between releases and reverted by a person who never opened the model’s code.

The gold set, and the ceiling under it

Promotion needs something to measure against. Yegge’s answer is a gold set of “between 200 and 1000 real cases” per intent. He sizes it by stakes: 1000 for money intents, 200 to 300 for FAQs.

Two details in that sentence carry weight. The cases are real, drawn from the work the model will do. And the size scales with what a wrong answer costs, which puts the most human scoring effort where errors are most expensive.

Then Yegge sets a limit. “Two human scorers usually top out at agreeing around 90% of the time,” he writes, and that ceiling is also the ceiling on what evals can show.

The consequence is practical. If two human scorers agree on roughly nine cases in ten, then the reference itself is uncertain on about one case in ten. A model score above that line measures agreement with one scorer’s habits more than it measures correctness. Any promotion rule that demands near-perfect eval scores is asking the gold set for a precision it does not have.

For a team building its first gold set, this changes the first task. Measure how often your own scorers agree before you set any promotion threshold. The agreement rate tells you how high a threshold can meaningfully go.

A mapping your risk team already knows

Yegge draws the line to an existing discipline himself: “The policy table is the model inventory. The gold-set eval is the validation. The promotion sign-offs are the approvals.”

Yegge maps these artifacts onto model risk management. The mapping matters for adoption. In our reading, a regulated firm with a model risk function would have no need for a new governance category to accept an agent on its service desk. It could place the agent inside a review process that function already knows, with a named owner for each artifact.

It also gives the table a second audience. The service team reads it to know what the model may do. The risk team reads the same rows as an inventory with validation evidence and signed approvals attached. One document can serve both, which should keep the two views from drifting apart.

Writes stay with the orchestrator

The second constraint in the design concerns access. “The clerk will never see a credential, and it can never write directly to a system of record,” Yegge writes. In Yegge’s design, a mechanical orchestrator executes every write.

This separation is what makes a tier enforceable. A tier in a table is only a promise until something outside the model checks it before acting. In our design reading, the orchestrator is the natural place for that check: if it consults the policy table before each write, it can refuse an action the intent has not earned. The model holds no credential it could use to go around it.

For the permission side of support agents, see our piece on support agent authorization. The Counting House takes the direct write path away from the model entirely.

Why this matters for cheaper models

Put the pieces together and a pattern appears. The model starts with nothing. It earns each tier per intent against a human-scored reference. A separate orchestrator re-checks eligibility against the live system of record before it executes a write. Under those conditions, the question of which model to run becomes an empirical one, settled row by row in the table.

That is the lever. A cheaper model that clears the gold set for an FAQ intent can hold that tier, while a more capable model, or a person, keeps the refund queue. Cost and risk get decided per intent, with evidence attached, instead of once for the whole deployment.

Yegge adds a warning about who owns the evidence: “If you leave evals to a vendor then you’re locked in.” The gold set is the asset that lets a team swap models and re-test. In our reading, when the vendor holds it, every promotion and every model change depends on that vendor’s scoring. Owning the gold set keeps the model choice open.

For Yegge’s earlier argument about engineering, see The Dead Companies Walking. The Counting House moves his attention to casework, where the unit of trust is an intent.

What the evidence supports

The design is well argued. The field evidence is thin, and Yegge says so. “The one shop I’ve talked to about this is having early success with relatively inexpensive models.” That is one organization, reported by the author, and the sentence carries no number.

The essay also contains confusion-matrix numbers. They are illustrative examples of how the scoring works, and should be read that way. They are not results from a deployment.

Yegge sells workshops on this design, and the page discloses it. That does not weaken the logic of the policy table, which stands on its own. It does mean the case for adoption today rests on the reasoning more than on measured outcomes.

Do this now

Pick the three highest-volume intents in your support queue. For each one, write a row: the intent, its current autonomy tier (every row starts with no autonomy), the gold set behind it and the person who signs a promotion.

Then build one gold set before anything else. Pull 200 real cases for your most common FAQ intent, have two human scorers grade them independently and record how often they agree. That number is your ceiling. Set the promotion threshold below it, and keep the gold set in your own repository so the next model you test faces the same cases.

If your current agent can write to a system of record directly, move that write behind a component that reads the table first. Until that happens, the tiers in your table describe intentions, and only the orchestrator can turn them into controls.

---

This analysis synthesizes The Counting House (Steve Yegge, October 2026).

Victorino Group helps service and operations teams build policy tables, gold sets and eval gates that make AI agents auditable. Let’s talk.