The Judge Failed in Production. The Upgrade Lowered the Error by Not Trying. The Guardrail Worked.

A Vistaly beta customer looked at her first generated opportunity solution tree and said: “That is a lot of first layer opportunities there. I wish I could click on this card and be like, Clean this up.” Teresa Torres took that single complaint and ran sixteen experiment variants against it, publishing recall and specificity along the way. The write-up is on Product Talk, dated September 2026 in its page metadata.

One caveat before any number. This is one source, one product, written by the person who sells that product. The percentages are relative reductions on her own error counts, and the absolute counts are not all on the page. Read the figures as one team’s measured trajectory, then check whether the shape repeats in your own agent.

Three findings transfer to any agent that generates structure. A judge calibrated on a calibration set is not calibrated on production. A model upgrade can lower an error rate by declining to do the work. And the prompt-only variants traded one failure mode for another until a deterministic auditor was wired into the loop.

Four evals, and only one of them is code

Torres built four evals for the complaint. Tree-shape is deterministic: code assertions on node count, breadth violations, single-child count, children per parent, mid-level parents, max depth and CKM count. The other three are LLM judges: missed-groupings, poorly-framed-CKM, and parent-restaters, the last one binary.

That split decides everything that follows. Deterministic assertions have no calibration to drift. Two of the three judges turned out to need a second calibration, and much of the sixteen-variant story is learning what those judges were actually measuring.

The judge was calibrated. Production disagreed.

The missed-groupings judge started at 100% recall and 43.75% specificity. That specificity means it flagged more than half of the clean cases. Seven iterations followed: simplify the prompt, extend it, add more examples, and, in Torres’s words, “bumped to a smarter model”. After seven rounds it sat at 100% recall and 60% specificity. Sixteen points of specificity, the gap between 43.75% and 60%, for seven rounds of work on the judge itself.

The root cause was elsewhere. The judge was being confused by unrelated upstream errors in the trees it was grading. Hamel Husain’s advice had been to fix the upstream errors first, advice Torres says she initially resisted. Once those were fixed, the parent-restaters judge reached 100% recall and 93% specificity, and the poorly-framed-CKM judge reached 90% recall and 100% specificity. Calibrated, by any reasonable standard.

Variant 1 ran on production data. The parent-restaters and poorly-framed-CKM evals saturated above 90% error in both the variant and the control. A judge that finds more than 90% error on both arms of an experiment is reporting on itself. The judges, calibrated on a calibration set, were miscalibrated against what the product actually produced. Torres’s first lesson is the direct one: recalibrate judges against production samples.

We covered golden sets, per-criterion judges and consensus scoring in Netflix’s 600-synopsis golden set. Torres’s numbers show what comes after the golden set passes: the same judge, moved onto production traffic, needs a second calibration, and the failure announces itself as an implausibly high error rate on the control arm.

The model upgrade lowered the error by not doing the work

Variant 7 swapped Sonnet 4.6 for Sonnet 5. Parent-restater errors dropped to zero. On the parent-restaters metric alone, the upgrade looked like a clean win.

The mid-level parent counts told a different story. In one case where Sonnet 4.6 had created 6 mid-level parents, Sonnet 5 created 1. In another, 4.6 had created 11 and Sonnet 5 created 1. The error fell because the groupings were not attempted. A parent that does not exist cannot restate its children.

An error rate without a companion count of attempts will reward this every time. Mid-level parents is one of the tree-shape assertions, and that count is where the trade shows. Any team gating a model upgrade on a single error metric should assume this trade is on the table and check the attempt count before shipping.

Prompt variants, and one trade after another

Cumulatively across variants 1 through 8, parent-restaters fell by about 70% and poorly-framed CKMs by about 54%. That is the combined effect of eight variants, and a reader who wants a single winning variant will not find one in that figure.

The variants that lived in the prompt kept trading. Fixing upstream errors first is Torres’s second lesson, and she notes that it can worsen the downstream error. The prompt-only variants moved the error around. Her third lesson follows from that experience: prompt engineering, context engineering, orchestration and guardrails have to be worked together.

From eval to guardrail

Variant 16 changed the position of the check. Torres wired a deterministic audit tool into the agent’s loop, alongside a fix for a one-child overcorrection. After each generation step, the auditor returns its findings to the model for self-correction. She chose a deterministic tool specifically to avoid adding another LLM call. Her section title for this step is “Moving from Eval to Guardrail.”

The result on her own error counts: a 78% reduction in parent-restaters, a 29% reduction in poorly-framed-CKMs, and a 65% increase in mid-level parents. All three numbers moved the right way at once. The mid-level parent count is the one that matters most here, because it is the count that collapsed to 1 in the model-swap test.

The eval measures the tree after it exists. The auditor runs inside the loop and hands the model its findings while there is still a step left to act on them. Same kind of deterministic check, different position in the pipeline, and the second position is the one that acts as a control. We described layered detection in the abstract in Your AI Will Hallucinate. Build the System That Catches It. and argued that evaluation is governance in Evaluation-Driven Development. Torres’s variant 16 is the concrete version of both: deterministic checks promoted from a report into a step inside the loop.

Do this now

Pick one LLM judge in your agent and score a sample of production outputs with it. If the error rate lands above 90% on outputs you consider acceptable, the judge is miscalibrated, and every experiment you have run through it is suspect. Recalibrate on production samples before touching the agent.

Next to every error rate you track, place a count of attempts. For the tree, that number is mid-level parents. For your agent, it is whatever the model can skip to make the error disappear. Gate any model swap on both numbers moving in the right direction.

Find the eval you already run as deterministic code. Move it into the loop as an auditor that returns findings to the model for self-correction after each step. No extra LLM call, no new judge to calibrate. Torres’s variant 16 result came from a deterministic tool placed where the model still had a turn to fix what it found, together with the one-child overcorrection fix.

Fix upstream errors first and expect the downstream metric to get worse for a while. Plan the sequence and tell the people reading the dashboard before it happens. If your PM rubric for agent judgment still lacks a line for this, the rubric we covered is a starting point.

---

This analysis synthesizes 4 New Evals and 16 Experiment Variants to Fix 1 Customer Complaint (Product Talk, Teresa Torres, September 2026).

Victorino Group helps engineering teams move agent evals from after-the-fact reports into deterministic guardrails inside the generation loop. Let’s talk.