They Wanted to Pay $20 a Review, and Used More Agents to Produce Fewer Comments

The AI code review market has a price it is converging on, and Wealthfront wrote it down in order to walk away from it: “a unit price of around $0.25 per review and 2-3 minutes.” Their stated position was the inversion. “We wanted to pay $20 per review and were happy to wait 15 minutes or more.”

The system they shipped, called Iris, currently averages ten minutes and $4 per review. That is sixteen times the market unit price and a fifth of what they said they were willing to spend. Both numbers matter. The $20 is a declared budget ceiling, an internal permission to spend. The $4 is what the architecture actually consumed once it was built. Teams that quote the $20 as a cost benchmark are quoting an intention.

What they bought with that permission is the interesting part. It was not a bigger model, and it was not more findings.

The budget went to an adversarial split

Iris runs three models in defined roles. Opus 4 sits as an impartial judge. GPT-5 is described as “the highly-paid prosecution,” prompted to “investigate and find evidence of the problem.” Gemini 3.0 Flash is “the cheap defense,” prompted to “find evidence this isn’t a problem.”

The prosecution prompt carries an explicit release valve: “null results are expected and fine.” That single clause is the load-bearing piece of the whole design, and the reason is stated plainly by Austin McKee in the writeup: “LLMs are strongly biased to do what you ask. If you ask an LLM to find problems, it finds problems, even if it must make them up.”

Most review pipelines are one model told to find problems. The output of such a pipeline is not a measurement of the code. It is a measurement of how hard you asked. Wealthfront spent their budget building an opponent whose job is to make the finding survive contact with a counter-argument, and a judge who never had a stake in either side.

Asymmetric pricing is part of the design. Prosecution is expensive because manufacturing a plausible defect is easy and demonstrating a real one is hard. Defense runs on a cheap fast model because refuting a fabricated finding usually only requires reading the surrounding code.

The rubric is the real artifact

Wealthfront published a human rating scale, one through five. Level 1 is “actively harmful feedback.” Level 2 is “annoying, weird response.” Both carry the same gloss: “I regret having to read this bot feedback.” Level 5 is “The bot helped prevent a bug from being merged.”

Two of the five levels are explicitly negative value. Not neutral, not low signal. Negative. A rubric that admits its tool can score below zero is a different instrument from a benchmark that counts findings, because it forces the team to subtract.

We wrote in February that every vendor wins its own test, and in April that nobody was reviewing the reviewer. This rubric is the first published attempt I have seen to make reviewer trust a scored, tracked object owned by the buyer rather than a satisfaction claim owned by the vendor. It is cheap to copy. Any team running an AI reviewer today could put those five levels in a spreadsheet this week and start rating comments.

From a bell curve to stairs

The reported result of the adversarial rework is a distribution change. Ratings moved from a bell curve centered on 3 to what the team describes as “stairs”: a small increase in fours and fives, and “a dramatic decrease in the number of ones and twos.” Most pull requests now receive no comments at all, and few receive more than one.

Wealthfront published those distributions as charts without numeric percentages, so the shape is the claim and the magnitude is not available. Take it qualitatively.

The direction is what an engineering leader should notice. The measured payoff of running three frontier models against every diff was not a rise in bugs caught. It was the near-elimination of comments that made engineers regret opening the PR. Trust in a reviewer is destroyed by its worst output, not built by its best, and the arithmetic is brutal: one confident wrong comment teaches an engineer to skim the next ten, including the correct one.

More agents, fewer comments. That is the trade the money bought.

Fewer tools, more filesystem

The other design reversal is about tooling. “Dozens of bespoke tools to retrieve data (read: MCP) inevitably lead to poor tool calling performance and context bloat.” Their answer was to collapse the retrieval surface into a sandboxed filesystem with a single read_file tool and a shell.

This runs against the reflex of most 2026 agent architectures, which treat every new capability as a new MCP server. At Wealthfront’s scale the reflex breaks: over 20 million lines of code, the majority of it tests, across more than 350 repositories. One of their directed-research agents carries roughly 80 engineer-written quality checks. Feeding that through dozens of narrow tools costs context the model needs for reasoning.

The reviewer is deliberately not a gate

Iris fires when the author clicks “Ready for Review,” and stays advisory. In their words, “the peer review is blocking but the AI review is not.”

That choice is easy to read as caution and worth reading as governance. A non-blocking reviewer that produces a bad comment costs one engineer thirty seconds of annoyance. A blocking reviewer that produces the same comment costs a negotiation with a machine that cannot be argued with. Keeping the human review as the bottleneck means the accountable reviewer stays a person, which is precisely the question we said most teams were answering by default.

It also protects the measurement. Comments an engineer can freely ignore generate honest ratings. Comments that hold the merge hostage generate compliance.

Do this now

Pick your five levels and start rating. Define what negative value looks like for your team, in your codebase, in the words your engineers actually use when a bot comment wastes their time. Rate a week of AI review comments against it. If levels 1 and 2 are more than a small minority of your volume, your reviewer is spending trust faster than it earns it, and no increase in detection rate will fix that.

Then check what your reviewer is prompted to do. If the instruction is some version of “find problems in this diff,” you have a machine optimized to satisfy that instruction. Give it an opponent, or at minimum give it explicit permission to return nothing.

---

This analysis synthesizes Experiments with AI Code Review (Wealthfront Engineering, August 2026), cross-referenced with our previous coverage of who reviews the reviewer, BugBot’s evolution, and the benchmark paradox.

Victorino Group helps engineering teams define what their AI reviewers are allowed to cost them in trust, and measure it. Let’s talk.