Cockroach Labs Audits the Reviewer Before Merge. Copy the Gate Before You Turn On Auto-Merge

Between April 21 and September 11, an agent pipeline at Cockroach Labs merged 1,238 pull requests into a mirror repository it uses as a proving ground. After the first week of autonomous operation, the authors turned on what they call “human approval mode,” and only in that mode could they be certain the code was good enough to ship to customers. Seven of those merged PRs were later reverted. 92 issues were escalated to the human Chief. The token bill came to just over $135,000, which the authors put at around $84 per GitHub issue.

Those numbers come from a post by Adam Storm and Rafi Shamim describing five months of running bugs through a pipeline modeled on a hospital. Issues are patients. Agents are the medical team. A human sits at the top as the Chief. The interesting part is the last station before merge, which they call the Discharge Nurse.

The Discharge Nurse does something we rarely see in agent pipelines. In the authors’ words, “The Discharge Nurse audits the reviewer. The last gate before merge does not re-review the code. It verifies that the review happened properly.”

We have already written about teams that review less by design and about verification capacity as the binding constraint. This piece builds forward from there. If review is going to be done by agents, at volume, the control you need is evidence that the review happened and converged. We take three artifacts from the Cockroach post that, in our reading, produce that evidence, and two other practitioners explain why the first one matters and how to get there.

A gate that checks the review, with the code left to the reviewers

A second code review by another agent adds another opinion. An audit of the review asks a different question: did the process that was supposed to run actually run? Were the required reviewers present? Did their blocking findings get answered? Did the loop end because the work converged, or because something gave up?

In our view, that question is harder to fake than a fresh review, and a deterministic check can verify much of it. A model can produce a plausible approval for code it barely read. Producing a complete, consistent trail of resolved findings and recorded sign-offs is a much higher bar.

The Cockroach authors framed their own success criterion in the same spirit: “the question we cared about was not how many PRs per hour? but rather, how many of those PRs would we have been embarrassed to have merged ourselves?” One partial proxy in their data is seven reverts among 1,238 merged PRs. Two caveats apply. The figures come from a mirror of the repo used as a proving ground, separate from CockroachDB’s main repository, and a revert is a lagging, partial signal of embarrassment. The post also reports a separate Db2 run with striking speed and cost numbers. The authors say they were not reviewing any of that code and are still confirming correctness, so we leave those numbers out.

Why another agent’s review is weaker evidence than it looks

Cindy Sridharan, writing about debugging code you no longer understand, gives the clearest account of why stacking adversarial reviewers does not buy what teams expect. Her verdict on adversarial review: “an adversarial review cannot be trusted to work as instrument of verification. It works better as an instrument of elimination.”

Elimination still pays. In one of her own incidents, “about a third of the claims I challenged could not be backed up by citations and were culled.” That is a real cleanup. But a challenge that removes unsupported claims leaves the surviving ones unproven. Something else has to confirm them, such as a test or a human. Her figures come from her own small set of bugs and should be read as one practitioner’s account, and in one passage she warns against drawing too many conclusions “from this solitary example.”

She also proposes a metric we think belongs on every agent pipeline dashboard: “The number of human interventions per bug is the most honest measure of how agent-native a factory really is.”

Cockroach Labs reports one input to that measure: over the period, 92 issues reached the human Chief. That count leaves out the human approvals on merges, so it only gives a floor for that measure. We will not compute a ratio from it. The point is that the count exists, and a pipeline that counts its escalations has started on that measure.

The precedent log only a human can write

The second artifact, as we read the post, is a precedent log only a human can write. Here is how we read the principle: when a human settles an ambiguous case, the decision is recorded as precedent, and agents can use it without being able to change it.

This matters because the alternative is drift. An agent that can write its own precedents can, over many rounds, rewrite the standard it is judged against. Each change looks reasonable in isolation. The cumulative effect is a review bar set by the reviewee. A write-only-by-humans log turns every judgment call into an auditable decision with an owner, and it keeps the definition of “acceptable” in human hands while the volume of work sits with agents.

The same team also audited the instructions themselves. In July they reviewed 25 skill files totaling a little over 100,000 words and flagged about 23,000 words, roughly 23%, as removable. Instruction text accumulates the way precedent can. Someone has to own pruning it, and that someone should have the authority agents lack.

The written rule for when to stop

The third artifact is a circuit-breaker. Cockroach Labs hands a PR off once it “has gone four consecutive rework rounds that each surfaced a new blocking finding, or six substantive rounds of any kind.”

Two things make this rule useful. It is written down, with numbers, before the loop starts. And it distinguishes between a loop that is converging (findings shrink) and one that is churning (each round surfaces something new and blocking). Without a rule like this, an agent review loop has two failure modes: it runs indefinitely and burns tokens, or it ends when the reviewer gets tired of finding things, which looks identical to convergence in the logs.

The authors are candid that the rule “merged rather recently” and that they “don’t have good data to point to its efficacy.” We read that as a reason to adopt the rule and instrument it, rather than a reason to wait. A written stop condition that you measure is better evidence than an unwritten one that you cannot.

Encode the review before you automate the approval

Swizec Teller, writing from a 21-person engineering organization, describes where this lands in practice. His team auto-approves and merges “16% of all pull requests.” Getting there involved “a few dozen custom linters,” which encode review judgments into checks that run the same way every time. The team also rejects “half the comments BugBot makes,” so even a review bot’s output still passes through a human filter.

The order is the lesson. Swizec’s team turned recurring review judgments into deterministic checks first, then let a bounded slice of PRs merge on those checks. His numbers are his own operating figures with no published method, and a 21-person org is not a template for every company. The sequence still transfers: you earn automated approval by first making the review legible to a machine.

That connects back to an argument we made about prompted testing technique. An instruction to the agent is guidance. A check that runs regardless of what the agent decided is a control. A linter is a control. An audit of the review trail is a control. A reminder in the system prompt to “review carefully” is guidance, like any other instruction.

Do this now

If agents already review code in your organization, or you plan to switch on auto-merge this quarter, put three artifacts in place first. Cockroach turned on human approval mode after its first week. The audit gate is what we would require before turning that mode off.

A pre-merge audit of the review. Define what a complete review trail looks like for your repo: which reviewers, which findings must be resolved, which sign-offs recorded. Add a gate that fails the merge when the trail is incomplete. It does not need to read the diff.
A precedent log with human-only write access. Every escalated judgment call gets written down by the human who made it. Agents read it. Enforce the write restriction with permissions, not with an instruction.
A written loop-stop rule with numbers. Start from Cockroach’s four consecutive rounds with new blocking findings, or six substantive rounds total, and tune from your own data. Log every trip of the breaker.

Then track human interventions per issue, as Sridharan suggests, and count reverts of merged agent PRs. Those two numbers will tell you more about whether your pipeline is trustworthy than any throughput chart.

This analysis synthesizes Five months treating bugs like patients and coding agents like a medical team (Cockroach Labs, October 2026), How to effectively debug when you no longer understand the code (Cindy Sridharan, October 2026), and When code is cheap, judgement becomes the job (Swizec Teller, October 2026).

Victorino Group helps engineering teams put audit gates and stop rules in place so agent auto-merge is defensible. Let’s talk.