1,393 Agents Refactored a Million-Line Codebase. The Tests Missed 65 Regressions. Review Did.

Two essays published in September 2026 argue that human code review is on its way out. Allen Hutchison’s title states the position: The Test Suite Is the New Code Review. An unsigned post on minid.net goes further: “Human code review is not a fundamental property of software engineering. It is a historical approximation to a problem we did not know how to solve any better.”

The same month, Nous Research published the largest swarm refactor I have seen written up with numbers: 1,393 subagents rewriting the Hermes codebase, with the test suite as one of its gates. Two rounds of community review then found two classes of regression the existing tests had not caught. The write-up is worth reading as a controlled test of the essays’ claims, run by a team that published what review found afterwards.

The run, on its own numbers

Per the Nous write-up, the run used 1,393 subagents over about 19 active hours, with a peak of 218 running concurrently, 36 non-overlapping groups, and three levels of sub-delegation. Inference ran on Claude Fable 5.1. The run started September 2 and the PR merged September 4. The prompt, as quoted on the page, reads: “No excuses. No waiting for my decisions. Get it all done, and present me a PR or set of PRs when done.”

The output is real. Non-test Python went from 1,063,826 lines to 698,363, a 34.4% cut. Files over 5,000 lines dropped from 37 to 6. Functions over 300 lines dropped from 192 to 2. The longest if/elif chain went from 92 branches to 9. The single file gateway/run.py shrank from 34,847 lines to 5,512.

Two hedges from the same page belong next to those figures. Six files still exceed 5,000 lines. Module count and import dependencies rose.

The cost line: about $19,300 in model cost for the main run, about $25,000 including follow-ups. Nous’s own comparison for a small human team doing the same work runs from $150,000 to $1.8 million over two months to two years. Then a sentence the essays would need to answer: “That excludes human review time.”

Two classes of regression the tests did not catch

The existing tests had not caught them when the PR went up. Community review then found two things.

First, removed public names. In Nous’s words: “Reviewers caught public names that workers had removed because they had no callers inside the repository, even though external plugins could import them.” My reading of the mechanism: a worker looked for callers, found none inside the tree, and deleted the symbol. The tests agreed, because tests live inside the tree too. The consumers who could have broken were outside the repository, invisible to both the agent and the suite.

Second, exception handling. “An automated rewrite of suppress() calls also changed exception handling at roughly 65 sites. These were real regressions the existing tests had missed.” A mechanical rewrite applied uniformly, wrong at about 65 sites, and the existing tests had caught none of them. The page notes that further fixes followed after merge.

Both classes share a shape. The test suite encodes the behaviour someone already thought to assert. A public interface with no in-tree caller, and an exception path with no test, are exactly the places nobody had asserted anything. A swarm that optimises against the suite will find those places, because they are where the suite offers no resistance.

Hutchison and Minid are right that other practices can absorb much of what review does today. We argued the same thing in Your Code Review Is One Control Doing Five Jobs, and I will not re-derive the list here. The Nous data adds the part that essay could only assert: after four of those jobs move to pairing, design sessions and automated checks, what remains is the boundary and blast-radius reading that caught these two classes.

The three controls the essays leave out

The more useful content in the Nous write-up is the set of controls the run used and neither essay, as read, describes.

A frozen baseline. When a worker hit a failure, the instruction was to reproduce on origin/main HEAD in a clean environment to check whether the failure was pre-existing. Without this, a swarm attributes every red test it meets to its own change and starts fixing things it did not break, or the reverse, marking its own breakage as inherited. The baseline is what makes a passing run a statement about the diff rather than about the repository.

Interface preservation diffs. A tool’s JSON schema had to be identical before and after. A CLI’s help output was compared byte for byte. This is a separate gate from the test suite, aimed at a different question: did the shape of what we expose change? The write-up describes it for tool schemas and CLI surfaces. The removed Python names were found by reviewers, which suggests the same check, extended to public symbols, would have caught them earlier.

Isolation and checkpoints. Each worker ran in its own git worktree and committed after each verified step. That bounds the blast radius of a bad step to one worktree and one commit.

None of these are new ideas. Together they are the difference between “we let the agents run against the suite” and a run whose failures were locatable afterwards. Hutchison argues for the suite as the gate. Minid argues review was never fundamental. The run that actually shipped used the suite plus a baseline, plus an interface diff, plus per-worker isolation, plus human review, and still merged with regressions.

The cost line that excludes review

Hutchison’s own evidence is one repository split. A pull request in the extracted groups repo clears CI in about two minutes, against thirteen in the monorepo, his single-repo before and after. His heuristic for what deserves its own repo: “one clear job, one owner, and an API narrow enough to describe in a sentence.” That is a sensible rule and a sample of one. Minid’s post carries no data, so it stands only as the stated position.

The Nous page has a benchmark of its own, and it deserves the same scrutiny. Symbol lookups improved: average tokens per lookup fell from 2,218 to 993, and lookups needing a second read window fell from 628 to 184 out of 4,000. The median lookup actually returned more tokens. The average fell because the largest definitions shrank. Nous is explicit about the scope: “we didn’t measure agents completing engineering tasks.” The refactor made the code cheaper to look up. Whether it made agents better at finishing work is unmeasured.

So the numbers that matter for a team deciding how to gate agent output are these. Model cost about $19,300. Human review time excluded from that figure. About 65 regressions plus an unstated number of removed public names, missed by the existing tests and found by human review, with more fixes after merge. Anyone quoting the cost line as the price of the refactor is quoting the part that was measured.

Do this now

If agents are writing into your repositories this quarter, take three actions before you adopt Hutchison’s position that the suite is the gate as policy.

Add an interface diff to CI, separate from the suite. Export every public symbol, tool schema, CLI help text and API route before and after the change, and diff them. Any removal is a blocking finding until a human confirms there are no external consumers. This is the control that catches the first Nous regression class.
Freeze the baseline for every swarm run. Before the run, record the suite result on a clean checkout of main. During the run, every failure gets reproduced against that baseline first. A worker is not allowed to fix a failure it cannot attribute to its own diff.
Budget human review as a line item and route it by risk. Nous excluded review time from the $19,300. Do the opposite: log it. Then narrow it. Mechanical rewrites applied at scale (the suppress() case) and anything that removes a public name go to a human. Everything the interface diff and the suite cover cleanly does not. Where that gate sits in the write path is the question we mapped in Four Accounts of Agent Write Access, and the bar for what counts as passing should come from your own merged PRs, not from a vendor’s benchmark.

Hutchison predicts a future where the suite replaces review. The largest run I have read about this month used the suite, four other controls, and two rounds of review, and still needed fixes after merge. Plan for that run, not for the prediction.

This analysis synthesizes Refactoring Hermes with 1,393 agents (Nous Research, September 2026), The Test Suite Is the New Code Review (Allen Hutchison, Vycari, September 2026), and Three +1s and a prayer (minid.net, September 2026).

Victorino Group helps engineering teams design the gates that sit between agent output and the main branch: interface diffs, frozen baselines and risk-routed review. Let’s talk.