A Harness Took the Cheap Model From 3 to 10 Accepted Builds. The Frontier Model Stayed at 10.

GPT-6 Luna, run without a harness, produced 3 acceptable builds out of 10. With a behavioral harness wrapped around it, the same model produced 10 out of 10. GPT-6 Sol, the stronger model, was accepted in all 10 runs with the harness and in all 10 without it.

Those numbers come from Naresh Jain’s Are Harnesses Worth the Cost?, published on martinfowler.com. The setup is narrow and stated plainly: “10 paired runs per model on 1 greenfield Java service”, both models “at low reasoning effort”. A build counted as acceptable when it “started successfully and passed all 178 external acceptance tests”.

The finding we care about sits in the cost column. A harnessed Luna attempt cost about $0.273. An unharnessed Sol attempt cost about $0.406. In Jain’s words, “a harnessed Luna attempt cost about 33% less than an unharnessed Sol attempt”, and he qualifies it as true “in this sample”.

The four rows

The full results table, averaged over 10 runs per condition:

Condition	Tokens	Time	Est. cost	Accepted
Sol, harnessed	11,331,970	19 min 53 s	~$2.679	10 of 10
Sol, unharnessed	1,076,371	6 min 51 s	~$0.406	10 of 10
Luna, harnessed	22,933,648	35 min	~$0.273	10 of 10
Luna, unharnessed	5,860,736	15 min 56 s	~$0.086	3 of 10

For Sol, the harness bought nothing on acceptance. Jain puts it directly: “the harness used about 10.5 times as many tokens, took nearly 3 times as long, and cost about 7 times more, yet both conditions succeeded in all 10 runs.”

For Luna, the harness added “an average of 17,072,912 tokens, 19 minutes 4 seconds of autonomous runtime, and about $0.187 of estimated API-equivalent cost per attempt”. That spend moved the acceptance count from 3 to 10.

So the same harness had two very different effects. On the strong model it was pure overhead in this sample. On the weak model it was the difference between a usable result and a coin toss weighted against you.

The failed runs were close

One detail changes how to read the 3 of 10. Jain reports that “unharnessed GPT-6 Luna runs passed 94.8% of the functional checks.” The failed runs “still passed at least 144 of the 178”.

An unharnessed Luna build that fails is mostly correct. That is the expensive kind of failure. A build that crashes on startup announces itself. A build that passes 144 of 178 acceptance tests looks finished until someone runs the other 34. With the harness in place, all 10 Luna runs cleared all 178.

This connects to an argument we made when Builder.io traced its gains to the verifier: the check decides what a smaller model can be trusted with. Jain’s data adds what that single case lacked, a paired comparison across two models with the same gate.

Cost per accepted result

Cost per attempt is the obvious number to read off this table, and it misleads here. The useful unit is cost per accepted result.

On API-equivalent compute alone, the arithmetic from Jain’s table runs roughly like this:

• Sol unharnessed: about $0.406 per accepted build, since every attempt was accepted.
• Luna harnessed: about $0.273 per accepted build, for the same reason.
• Luna unharnessed: about $0.86 spent across 10 attempts for 3 accepted builds, so roughly $0.29 per accepted build.
• Sol harnessed: about $2.679 per accepted build.

The third line looks competitive on compute. It hides the seven unacceptable builds, each of which someone has to find, diagnose and repair. Jain’s numbers exclude that repair work. They also exclude the work of building and maintaining the harness, so the harnessed rows carry their own unpriced bill. The difference is that the unharnessed Luna row looks competitive while hiding seven failures.

That is where Michael Cutler’s proposal in How to Measure the Human Cost of Agent Work fits. He defines two ratios: “human minutes per accepted task = all cohort human minutes / accepted tasks” and “agent spend per accepted task = all cohort agent spend / accepted tasks”. Both divide by accepted tasks. Jain’s table gives you the second ratio. Cutler’s ledger asks you to measure the first, and he warns that “an unrecorded review is not a zero-minute review.”

Put the two together and the decision about a harness stops being a debate about engineering taste. It becomes a measurement: agent spend plus human minutes, divided by results someone accepted.

What the sample does not tell you

Jain is careful with his own claims, and the caveats belong in any reading of the table.

The costs “are API-equivalent estimates based on recorded token use and model prices, not actual billed costs.” They “exclude the upfront effort of authoring, reviewing, and maintaining the contracts, examples, assertions, and harness. They also leave out the effort of finding, diagnosing, and repairing an unacceptable result.” And: “This small sample cannot tell us the long-run failure rate or the total cost of achieving an acceptable application.”

There is more to weigh. The experiment covers one Java service, 10 runs per model, and low reasoning effort only. Jain describes the brief as “a favorable starting point for an unaided agent”, so harder work could widen or narrow the difference. He founded Specmatic, the tool that built the harness, and he discloses that he used generative AI to research the topic and build parts of the harness. None of that invalidates the table. It does mean the result is one well-documented sample from an interested party, and you should treat it as a hypothesis to test on your own work.

Cutler’s piece carries its own label: “It is a proposed method, not a report of an experiment I have completed.” Its worked numbers are synthetic, so we cite only the method.

The harnessed Luna row also has a price that is not in dollars. It used about 21 times the tokens of unharnessed Sol and took about 5 times as long. If wall-clock time to a result matters more to you than API spend, the cheaper row may still be the wrong one.

The gate stays when the model improves

The harness gave Sol no acceptance benefit in this sample. A tempting conclusion follows: strong models do not need gates. Jain rejects it. His summary is that model capability changes when and how often sensors run, while the need for an independent verification gate remains.

That matches the reading we have taken in regulated agentic work and in governing the harness. A stronger model lets you run the checks less often or later in the loop. It does not give you a reason to have nobody checking. Sol passing 10 of 10 unaided in this sample says nothing about run 11, and the 178 acceptance tests are what told anyone that Sol passed at all.

The practical consequence is that the harness is something you tune per model. Against a strong model you might run the full gate once at the end. Against a cheaper model you run sensors throughout and let the model iterate against them. The gate is the same. The cadence changes.

Do this now

Pick one recurring agent task your team already runs. Count, for the last month, how many outputs were accepted and how many were sent back or rewritten. Record agent spend and, honestly, the human minutes spent reviewing and repairing, using Cutler’s two ratios.

Then run the cheaper model you have been avoiding through the strongest verification gate you can build for that task, for a small paired batch. Compare cost per accepted result, including human minutes and time to result. If the cheaper model with a gate wins on your numbers, you have found a cost lever. If it loses, you have a measured reason to keep paying for the frontier model.

---

This analysis synthesizes Are Harnesses Worth the Cost? (martinfowler.com, Naresh Jain, October 2026) and How to Measure the Human Cost of Agent Work (Cutler SG, Michael Cutler, October 2026).

Victorino Group helps engineering teams build the verification gates and cost ledgers that show which model a task actually needs. Let’s talk.