One Wrong Word in AGENTS.md Cut the Compile Rate from 90 to 70 in One Bench
A design-system bench run described by Southleft graded three contexts on the same ten design tasks, three repetitions each. The share of generations that compiled against the library’s real types was 90.0 with no instruction file, 70.0 with an AGENTS.md, and 76.7 with a skill. Of the 19 generations that failed hard across the run, the page traces 10 to a single word.
The AGENTS.md listed error as a status tone. The library’s canonical value is danger. In the bare context, zero generations hit the problem.
Southleft then rewrote the file as an index that points to a generated contract and reports a return to the same 90-point baseline, with a document 90% smaller than the original.
Read the result with the limits in front. It covers one design system, three repetitions per cell, no variance shown. The author sells design-system services, and the page carries a consultancy call to action. The page does not name the coding agent or the model. This is a pattern worth testing in your own setup, and the evidence does not go further than that.
What the bench measured
The page states the design as 3 different scenarios x 3 times each x 10 design tasks, 90 graded generations. The three scenarios map to the three contexts in its results table, 30 generations each:
| Context | Compile against real types | Hard failures (of 30) | Mean score | Judgment axis |
|---|---|---|---|---|
| Bare (no file) | 90.0 | 3 | 95.5 | 89.3 |
| AGENTS.md | 70.0 | 9 | 93.0 | 89.6 |
| Skill | 76.7 | 7 | 94.2 | 90.4 |
In this first run, no context beat the bare one on compile rate or on mean score. Both instruction contexts scored below bare on compile rate and mean score. The page’s explanation covers the AGENTS.md file, and the excerpts we read give no cause for the skill’s drop.
One wrong word
The source puts it in one sentence: “One wrong word in the steering document cost ten generations.” The file said error. The library’s canon is danger. Ten failed generations traced to that value.
The failure here is narrow, and it is specific. The page does not report that instruction files hurt in general, and nothing in its numbers supports reading it that way. One value was wrong, and it was wrong in a place the agent reads. The bare context had no such file, so it had no such failures.
Our reading of the mechanism: a file that restates vocabulary becomes a second source of truth next to the types. When the two agree, the copy adds nothing, since the types already say it. When they diverge, the copy wins the argument with the agent and loses it with the compiler. The bench shows one divergence, caused by one wrong value. It does not show drift over time, though the same defect is what staleness would produce later.
What the fix did
The page describes removing 90% of the AGENTS.md content and scoring lower, 72 against 90. The table gives 70.0 for the file as first written. The excerpts we have do not say whether the 72 is a separate run, so we use the table figure in the comparison above and flag the difference here.
Then the file was changed to point at the generated contract, and the score went back: “we sit at the same 90 points baseline with a document that is 90% smaller”. Southleft also reports test token cost down around 41% and time down around 22%. The page excerpts we read do not state the comparator for those two percentages, so we repeat them as the source’s figures and draw nothing from them.
The design principle on the page is worth quoting whole: “Don’t give agents a list of answers. Give them the language and principles they can use to arrive at the right answer.” A list of status tones is a list of answers. A pointer to the contract that defines them is language the agent can resolve at generation time, from the same artifact the compiler checks.
The judgment axis barely moved
Judgment scores were 89.3 bare, 89.6 with AGENTS.md, 90.4 with the skill. That is a spread of about one point, while the compile axis moved twenty. Whatever the instruction files were doing, this bench did not catch them improving judgment.
We read this against our own claim, and it should be stated as ours: an instruction file earns its place by carrying the judgment the types cannot encode (when to use a status tone, how a component should feel, what a team never ships). The Southleft bench gives no evidence for that half of the claim. It gives evidence for the other half: facts the types already hold do not belong in prose, because a wrong copy had a measurable cost here, and this bench did not test a correct one.
How this fits what we have published
Our post on passive context covered a Vercel evaluation in which a compressed AGENTS.md index outperformed skills. Southleft’s rewritten file is also an index, so it is consistent with that post and does not reverse it. Before the rewrite, the skill scored higher than the AGENTS.md in this bench (76.7 vs 70.0).
Our post on a design policy file and its eval loop covered Vercel’s three-way split: judgment goes in prose, reusable mechanics go in a stylesheet, and failures a machine can detect get a deterministic check. Southleft’s split is similar. Prose carries judgment, and the vocabulary comes from a generated contract. What the Southleft case adds is a narrower rule inside the index itself: a copied list of values is where the risk sits, and a pointer to the generated artifact removes it.
What the evidence cannot say
Four limits. First, three repetitions per cell with no variance shown; a 20-point swing on 30 generations is large, and a different draw could land differently. Second, one design system, so nothing here generalizes across libraries. Third, a vendor author with a service to sell, which is a reason to read the numbers carefully and a reason to check the bench yourself; it lives in a third-party repository that we did not read. Fourth, the accounting has a loose end: the page attributes 10 failures to the error value, while the AGENTS.md context shows 9 hard failures. Either one of the ten sits in the skill context (the page puts none in the bare context), or the two figures count different things. What we have does not say which.
The page also gives an external-readiness score of 40 to 50. That figure is the author’s own estimate, and we do not use it.
Do this now
Pick one instruction file that your agents read in a codebase with typed APIs. Search it for literal values: enum members, token names, prop names, status tones, function names. List each one.
For every item, find the artifact that already holds it: the generated types, a schema, a token export. Delete the copy and put a pointer in its place, or generate that section of the file from the artifact at build time so a person never types it.
Then run the check Southleft ran, at a size you can afford. Take ten real tasks. Run each three times with no file and three times with the file. Count how many generations compile against the real types. If the file scores below bare, find the values it states that the types also state. Start there.
This analysis synthesizes What Does AI-Readiness Mean in Design Systems? | Southleft (Southleft, September 2026).
Victorino Group helps engineering teams design the instruction files their agents read, keep them small, and measure whether they help before anyone trusts them. Let’s talk.
All articles on The Thinking Wire are written with the assistance of Anthropic's Opus LLM. Each piece goes through multi-agent research to verify facts and surface contradictions, followed by human review and approval before publication. If you find any inaccurate information or wish to contact our editorial team, please reach out at editorial@victorinollc.com . About The Thinking Wire →
If this resonates, let's talk
We help companies implement AI without losing control.
Schedule a Conversation