Builder.io Found Every Jump Came From the Check: What a Trustworthy Verifier Contains

An import job where “88% of its pixels didn’t match the original” fell to 2% over one weekend, using one smaller model. Builder.io reports this as a single job, self-reported, and credits the verifier for it: “Every jump between rounds came from changing the check, not the model.”

That sentence is the useful part. When a loop stalls, Builder.io writes, “it’s tempting to blame the model and switch to a bigger one. That wouldn’t have helped here. The check was wrong.” A second team, IronBee, describes the same discipline from the other side: a judge that can only pick from options it is offered and must cite evidence ids.

Both results come from one team each and both are self-reported. We do not read them as proof of any cost or accuracy claim. We read them as a rubric: what a check has to contain before you let a loop trust it. Related reading: three ingredients of production agent loops, the harness decides the score, self-improving harness and the eval set, the decision model on the control path. This piece builds forward from the construction of the verifier itself.

What a wrong check looks like

Builder.io reports four bugs in the check itself that made its results unreliable. Two of the numbers it gives show the shape of the problem.

Fonts that never loaded made one export “look 17% off when the real difference was 0.19%.” The check was measuring a rendering failure and reporting it as a design failure. In the other direction, a blank page scored 3 to 5% in the first version of the check and 57% or more after the fix. A blank page is a case where the right verdict is known, and the first version of the check got it wrong.

Notice what these failures have in common: neither is a model failure. In our reading, a stronger model pointed at this check would have chased a 17% error that did not exist, and capability spent against a wrong check is spent on the wrong target.

Calibrate the check on cases you already know

The blank page is the cheapest calibration case there is. You know the right answer before you run anything: it should score as a failure. The same holds for a known-good output, which should score as a pass.

We read Builder.io’s two numbers as an argument for a standing calibration step. Before a check judges any agent output, run it on a small set of cases where a human has already decided the verdict, including at least one that is obviously broken and one that is obviously fine. If the blank page passes, the check is not ready. In our reading, that step would have surfaced the 3 to 5% score on a blank page before it shaped a round.

The source does not say Builder.io ran calibration cases in advance. That is our recommendation, drawn from what the failures would have looked like if it had.

A holdout stops holding out once you tune on it

Builder.io’s round 4 passed 28 of 28 tracked cases and only 3 of 5 held-back cases. The stylesheet carried 21 media queries, many covering only 25 to 50 pixels. We read that as an output fitted to the cases it could see: narrow breakpoints are what you write when you are matching specific test widths rather than building a layout.

In our reading, the held-back cases did their job: they exposed the fit that the tracked cases rewarded.

IronBee’s result shows how quickly that protection can erode. Its 130-replay test ended as expected in 110 of 130 runs at first and in 130 of 130 after fixes. The fixes were tuned on the same test set, and the shop in the test was a localhost demo IronBee controls. This is a vendor result, and 130 of 130 on a set you tuned against says little about the next run on someone else’s site.

The practical rule is mechanical. A holdout is held back only while nobody looks at its failures. Once you read a held-back failure and adjust the agent or the check in response, that case has joined the tracked set. Keep a second, unread set for the next round, and refresh it.

A judge that picks and cites

IronBee’s design constraint is stated in five words: “Jev picks. It never writes.” The judge chooses among options it is handed. It does not compose a verdict from scratch. Each verdict cites evidence ids (IronBee shows [r3], [l12] and [s5] as examples), so a person can open the cited item and check whether it supports the call. An LLM steps in only as a fallback for stuck runs.

Two properties matter for a governed deployment, in our reading. A judge limited to offered options cannot invent a pass condition you did not list. And a verdict that points at an evidence id can be audited after the fact by someone who was not there.

IronBee also reports where its judge failed, which is more useful than the headline number. In one case it judged an Airbnb task done at p=0.99 when the search had not been re-run. In another, a Kindle order passed at p=0.51 while the real state was PAYMENT_FAILED. A high probability attached to a wrong verdict, and a pass attached to a failed payment, are the same defect from two angles: the score was not tied to the state of the world. This is why the evidence id matters more than the probability. A probability is the judge’s confidence in itself. An evidence id is something you can go and look at.

IronBee is a vendor case study that promotes Jev, which TypeSafe builds. We use it as a second data point for the construction pattern, not as a benchmark.

Agreement is not independence

A second reviewer added to a weak check can look like assurance. Philipp Dubach, in a conceptual piece on enterprise knowledge, states the limit directly: “Automated review can support this process, but agreement is not independent verification. Two agents may reach the same conclusion because they relied on the same mistaken decision note.”

Take the font bug as an illustration (ours, not Builder.io’s): two judges scoring the same broken render would both report the same wrong number. Independence comes from a different input, a different measurement or a human, and a second model reading the same artifact provides none of those.

One condition we would add

Neither source describes this next item, so treat it as our design proposal. A check the agent can edit is not a check. If the loop lets the agent loosen a threshold, skip a case or rewrite the scoring script to get a pass, the verifier is part of the thing being verified.

We would make the check off-limits to the agent, with an explicit escape. When the agent concludes the check is wrong (and sometimes it will be, as Builder.io’s four bugs show), it stops and writes down why instead of changing the check. A person then decides whether the check or the agent is at fault. The escape matters because without it an agent facing a broken check has two options, and one of them is to game it.

Do this now

Pick one loop you run today and audit its check against this list:

Run it on a blank or obviously broken output. Does it fail?
Run it on an output you know is correct. Does it pass?
Find the held-back set. Has anyone read its failures and acted on them? If yes, build a new one.
Look at a verdict. Does it cite something you can open, or only a score?
Check who can edit the check. If the agent can, remove that access and add a stop-and-explain path.

If the loop has stalled, do this audit before you change the model.

This analysis synthesizes Build an agent loop a small model can finish (Builder.io, October 2026), How We Built the Fastest, Cheapest Browser Agent with Jev (IronBee, October 2026), and Enterprise Knowledge Management: Connecting AI to Action (Philipp D. Dubach (personal blog), October 2026).

Victorino Group helps engineering teams audit the checks behind their agent loops before those checks decide what ships. Let’s talk.