Telling the Agent to Use TDD Is Not a Control. Dan Luu Ran 4,800 Times to Show It.

Dan Luu gave a coding agent the same task under thirty different testing instructions: use TDD, fuzz it, write a TLA+ model, load this vendor skill, make no mistakes. Each condition ran 80 times at two effort levels (medium and xhigh) on codex with GPT-5.6 Sol, which works out to roughly 4,800 runs. The metric was blunt: the fraction of runs that passed 100% of a hidden 37-test suite for a Zstd implementation in Rust. His summary of the graph: “nothing really wildly outperforms. However, Default (no additional instructions) does well above average.”

That sentence should reorganize how you think about the testing paragraph in your AGENTS.md. Naming a technique in the prompt did not beat saying nothing. In at least one condition it cost more and scored slightly lower.

We covered Luu’s August regex experiment in the holdout audit piece: a hidden test set as the control the agent cannot game. September’s experiment uses the same kind of holdout and asks a different question. Given a holdout, does the vocabulary in the prompt move the number? Across thirty conditions the answer is mostly no.

The instruction that did nothing

Start with the cleanest null result. One condition told the agent to “make no mistakes.” Luu’s read: “At every level at which I looked at the results, they were indistinguishable from random draws of Default.” At every level he looked at, nothing moved. The instruction was consumed and did nothing measurable.

That is the baseline for everything else in the study. A prompt line can be present, obeyed in the transcript, and inert on the outcome. When someone shows you a testing section in an agent config and says “we tell it to do X,” the burden is on them to show X moved a number against a holdout. Presence in the prompt is not evidence of effect.

TDD: obeyed, and worse

The TDD condition is the one I see most often in agent configs. Luu predicted it would underperform, and it did: “TDD didn’t do well, as predicted.” The interesting part is that the agents obeyed. TDD agents “had one or more failing tests in 67 of 160 cases before doing substantial (non-stub) implementation, vs. 0 of 160 for the Default condition.” They also “produced twice as many tests.”

So the instruction changed behaviour, visibly, in the direction the instruction asked for. Red tests first, more tests overall. And the holdout pass rate did not improve. Twice the tests, written by the same model that wrote the code, from the same reading of the spec, buys you twice the confirmation of whatever the model already believed. The hidden suite measured something the model’s own tests could not see.

This is the distinction that matters for anyone owning agent quality. A technique is a description of how a human organizes their own attention. TDD works for a person because the failing test forces a commitment to a behaviour before the implementation takes over. The agent does not share that failure mode in the same shape, so the ritual does not buy the same protection. What the agent needs is a check it did not author.

Formal methods: performed, not used

The TLA+ condition is the most vivid case of ritual without effect. Luu expected formal methods to underperform and they did, “but not for the reason I expected. Agents failed to use them remotely effectively, so of course they couldn’t outperform.” The numbers: “159/160 agents created some kind of TLA+ model,” and “I didn’t find an instance of a TLA+ issue resulting in an actual change in the Rust code.”

Nearly every run produced the artifact the instruction asked for. Zero runs, as far as the author could find, let the artifact change the code. The model was decoration. Anyone auditing those transcripts by checking “did the agent write a TLA+ spec?” would have marked 159 of 160 as compliant. Compliance with a technique and benefit from a technique are different measurements, and only the holdout sees the second one.

Fuzzing shows the same shape from the other side. Agents generated random structured inputs in “10 out of 160 cases,” and in those cases “this found real bugs half the time.” The technique works when applied. Random structured inputs appeared in ten runs of 160. Ten runs out of 160 is what an instruction looks like when the harness does not enforce it.

The vendor’s own skills made it worse

Here the study stops being an academic curiosity and starts describing a purchase decision. Luu ran the testing-related skills that codex itself recommended. “The testing-related skills codex recommended we try underperformed, although our quick custom skill did ok.”

The Hegel skill (34k characters plus a 45k reference, “more than 20k tokens”) produced slightly worse correctness at a cost that was “much higher (26% higher on medium and 41% on xhigh).” The Trail of Bits skill had a more basic problem: “only 108 out of 160 runs actually opened the skill to read it.” Roughly a third of the runs never read the skill.

Luu’s diagnosis: “the skills seemed written like they’re human tutorial instructions, in that the goal of the skill seems to be to explain how to do something.” The Hegel author, David R. MacIver, adds in the appendix a sentence worth keeping: “agents suck at writing agent skills and also everyone (including us) uses an agent to write their skills.”

We argued in the passive-context piece that an AGENTS.md file beats a skills library in evals. This experiment is a direct test of that claim on a vendor-recommended skill set, and the claim held. A skill written as a tutorial is a 20k-token tax that teaches a model something it either already knows or cannot apply from reading. The skill rot argument covers what happens to that tax over time. This study shows the tax on day one.

The five lines that worked

The highest-scoring condition was Luu’s own skill, five lines long. He is careful to call it “a first draft for a skill to iterate on” that “would need more than the 2 minutes I spent on it to be actually useful.” Take him at his word on the hedge. Also take his explanation of why five lines beat roughly 80k characters: “our skill is designed to nudge away from their default behavior towards more productive behaviors whereas the other skills seem more like tutorials.”

Steering versus teaching. A tutorial assumes the reader lacks knowledge. A steering instruction assumes the reader has the knowledge and a default that points the wrong way, and it names the alternative. The model already knows what the technique is. What it lacks is the impulse to apply it to this task. Five lines can supply the impulse. Tens of thousands of characters explaining the technique cannot, and in one skill’s case a third of the runs never opened the file.

The caveats belong here. One model, one vendor harness, one main task with a 37-test suite. A second task, an IMAP RFC implementation at 40 runs per condition, produced results that “weren’t materially different.” The author explicitly cautions against reading much into the ordering of conditions on the graph, and the page prints no per-condition percentages in text. The finding that survives the caveats is the null: across thirty ways of naming a technique, nothing “wildly outperforms,” in the author’s phrase, and the no-instruction baseline sat well above average.

What a control looks like

If naming the technique is not a control, the study also shows what is. Two things moved the number or exposed the failures, and neither lives in the prompt vocabulary.

The first is the hidden suite. Every result above exists only because 37 tests the agent never saw scored the runs. Without them, the TDD condition looks like the most diligent one (twice the tests, red before green) and the TLA+ condition looks like the most rigorous. The holdout is what separates the performed technique from the effective one. We have written about what agents do to a verification signal they can see. The corollary here is milder and more common: an agent does not need to game a visible test to be misled by one it wrote itself.

The second is enforced behaviour. Fuzzing found real bugs half the time it was applied, and it was applied in ten runs of 160. The prompt asked. The harness did not check. The distance between “the prompt says fuzz” and “the run fails if no fuzz target exists” is the distance between a request and a control. Luu’s five-line skill is a weak version of this: it did not enforce, it steered, and it still outscored everything that explained.

Do this now

Open the agent configuration for one repository and find the testing section. For each technique it names, answer two questions. Is there a hidden acceptance suite, owned by a human, that the agent never reads, against which this technique’s effect has been measured? Is there a harness check that fails the run when the technique is skipped? A line with two no answers is a “make no mistakes” line. It costs tokens and produces nothing you can see.

Then rewrite the section in the shape that won. Delete the explanations of what the technique is. Keep only the sentences that name a default the model has and point it somewhere else. If the result is under ten lines, that is consistent with the highest-scoring condition in this study. Measure it against the holdout before believing it, the same way Luu did.

This analysis synthesizes How well do agents use test and verification techniques? (Dan Luu, September 2026).

Victorino Group helps engineering teams replace prompt vocabulary with hidden acceptance suites and harness-enforced checks that agents cannot perform their way past. Let’s talk.