Review Was Never About Defects. That Is Why Automating It Costs So Much.

Alberto Bacchelli and Christian Bird sat with Microsoft developers, watched them review code, and counted what the comments were actually about. Defect-related comments made up only 14% of actual review comments. That result was published at ICSE in 2013 and it has been quietly ignored ever since, because the other 86% is hard to name and impossible to put on a dashboard.

Brian Houck, a Microsoft researcher writing in DX’s Engineering Enablement newsletter, revives that number at exactly the moment it matters. Every automated review product on the market is sold on defect detection. Defect detection was the smallest thing reviewers were doing.

What the Other 86% Was

In the same study, more than half of the developers said they use reviews to explore alternative solutions. Not to approve or reject an implementation, but to find out whether a different one existed. The rest of the comment volume is the ordinary business of an organization keeping track of itself: who owns this subsystem now, why this abstraction was chosen two years ago, which convention this team actually follows when the style guide is silent, whether the person who wrote this understood the constraint they were writing against.

Houck’s formulation is the cleanest one I have seen: “The visible output of code review is better code. The invisible output is a better engineering organization.”

The asymmetry between those two outputs is the whole problem. The visible one has a proxy metric that a tool can hill-climb. The invisible one has no metric at all, so when it degrades, nothing turns red. A team can lose its shared model of its own system for eighteen months and the only symptom will be that onboarding got slower and estimates got worse, neither of which anyone will attribute to review automation.

We have argued before that lost understanding is a real and invisible cost. The 14% figure gives that argument a denominator.

Meta Measured the Amplification

The pressure to automate is arithmetic in origin, and Meta published the arithmetic.

Over the past year, according to a Meta paper (arXiv 2605.30208) cited by Houck, significant lines of code per human-landed diff increased by 106%. Diffs per developer per month rose 51%. More than 80% of that growth came from agentic AI. So the same reviewers are now looking at roughly twice as much substantive code per change, across half again as many changes.

The queue responded the way queues do. Houck reports that the percentage of diffs reviewed within 24 hours is declining, and that in some large groups, reviewers are staring down thousands of pending reviews. DX’s own analysis found median pull request size grew by 64%.

Reviewers were already the constraint before any of this. A Microsoft study of 911 developers found that only 26% of developers said they always wrote a detailed description of the code being reviewed, and that the top three challenges were timely feedback, review size, and understanding the motivation for a change. A separate Microsoft study found developers ideally only want to spend about 7% of their time reviewing code. Agentic AI did not invent the bottleneck. It doubled the load on a process that was already running past its stated capacity, staffed by people who wanted to give it a fourteenth of their week.

RADAR Changes the Verb

Meta’s response is worth reading carefully, because it is not the product most vendors are selling.

RADAR stands for Risk Aware Diff Auto Review. Its reported results are strong: more than 535,000 diffs reviewed, over 331,000 landed, a revert rate roughly one-third that of non-RADAR diffs, a production incident rate one-fiftieth as high, and a 3.3x faster median time to close, roughly a 70% reduction.

The mechanism behind those numbers is eligibility, not review quality. RADAR’s job is to decide which diffs are low-risk enough to auto-approve, which means the human attention it frees up gets concentrated on the diffs that were never eligible. The revert rate is one-third of the baseline partly because the population RADAR handles was selected for being safe. That is a feature, and it is also the reason the number cannot be lifted out of context and used as a general claim about AI reviewing code.

Two caveats belong on this. First, Houck is citing his own institution’s research while publishing in DX’s newsletter, and DX sells developer-productivity measurement, so the framing that arrives with these numbers is not disinterested. Second, a comment thread on the post circulates a percentage that does not appear in the published body, likely from an earlier edit. Numbers that only exist in comments should not travel.

Degradation Arrives One Reasonable Decision at a Time

The failure mode here is procedural, and Houck describes it exactly: degradation arrives “through a series of individually reasonable decisions: this change is low risk, this review can be automated, this approval can be skipped.”

No one ever decides to stop understanding the system. Each step is defensible in isolation and the cumulative effect is invisible because the thing being lost was never measured. That is what makes it different from the usual quality-versus-speed trade, where at least one side of the trade shows up in a dashboard.

An anonymous developer in Microsoft’s “AI Where It Matters” research located the boundary in one sentence: “I can’t fully delegate the final code review to AI. My approval puts my name on it.” Accountability does not delegate. Whatever the tool concludes, a name goes on the merge, and that name belongs to someone who is expected to be able to explain the change six months later.

We have written that removing review renames governance rather than removing it. The 14% figure adds a second loss to that ledger: the organization’s own comprehension of the system it operates.

Do This Now: Stratify Before You Automate

The design answer available from the evidence is risk stratification with conservative eligibility. Concretely, in the next two weeks:

Write down your eligibility rule before you buy anything. Which classes of change may be auto-approved? Configuration in a non-production environment, dependency bumps within a pinned major, generated code with a deterministic generator, test-only additions. If your rule is “the AI reviewer says it looks fine,” you do not have an eligibility rule. You have a vendor’s default.

Make the ineligible set explicit and larger than feels comfortable. Auth paths, money paths, data deletion, schema migrations, anything touching a system where the last person who understood it has left. RADAR’s incident numbers come from the conservatism of the split, not from the model’s cleverness.

Instrument the invisible output. Pick two signals for organizational understanding and track them monthly: time for a new engineer to land a change in a given subsystem, and the count of subsystems with exactly one person who can explain them. If both worsen for two quarters while your review automation coverage rises, the 86% is what you traded away.

Keep a human on the merge for everything ineligible, and make the reviewer’s name the accountable one. Not as ceremony. As the mechanism that keeps at least one person obligated to understand the change.

Automating defect detection buys you the 14%. Deciding which code a human must still read is what protects the rest.

---

This analysis synthesizes What are code reviews even for? (DX / Engineering Enablement, August 2026).

Victorino Group helps engineering organizations design review eligibility rules that keep AI-scaled delivery inside a governed accountability boundary. Let’s talk.