Two Models One Point Apart Carry Opposite Regulatory Risk

On a 108-field extraction from a published European regulatory assessment report, Opus 4.8 had a higher omission rate and a lower fabrication rate than GLM 5.2. Both models sit inside a few points of each other on the overall benchmark. For a regulatory filing, those two failure modes are not equivalent in any sense that matters. An omitted field comes back as a deficiency letter. A fabricated field comes back as a data-integrity finding against the sponsor.

That single comparison comes from ClinReg, a benchmark published in July 2026 by Suchismita Padhy, Wenzhe Xue, Niklas Nielsen and Arjun Bansal at Everest / Log10. They ran 19 models (12 proprietary, 7 open-weight) three times each on real regulatory and clinical-trial tasks, taking the mean. The headline everyone repeated was the cost result: GLM 5.2 and Kimi K3 cost 33.8% and 59.6% of GPT 5.6 Sol per task on average, with GLM 5.2 running at roughly a tenth of Opus 5’s price. That number is real and boring. The interesting result is buried underneath it, and it changes how a regulated team should select a model.

Same Score, Opposite Failure

The top cluster is tight. GPT 5.6 Sol scored 88.4 with a standard deviation of 4.6. GLM 5.2 scored 87.4 with a standard deviation of 5.6. Kimi K3 scored 86.9 at 4.6. A second cluster (GPT 5.6 Terra, GPT 5.5, Grok 4.5, Opus 5) spans 84.3 to 86.3 with an 11x spread in cost. Once run-to-run variance is that large relative to the spacing between models, the ordering inside a cluster carries almost no information.

The authors say it directly: “Parity is not uniformity. Two models a point apart on the leaderboard are not interchangeable in production. Choose on error profile, not on rank.”

That instruction has teeth because the error profile is measurable and the rank is not. Their finding on where models actually diverge: “Models were broadly reliable at extracting information explicitly present in the source; the larger differences appeared when required fields were absent.” Every model reads well. What separates them is behavior at the edge of the evidence, where a required field has no source to support it and the model must decide between leaving a hole and filling one.

Two models can arrive at the same aggregate score by making the opposite choice there. A model that omits produces an incomplete submission that a reviewer catches. A model that fabricates produces a complete-looking submission that passes review and fails audit. In our earlier read on regulated harness engineering the lesson was that the control belongs in the harness, not the prompt. This is the measurement analogue. The metric belongs at the level of the failure you care about, not at the level of an average.

Check One: Split Your Score Into Fabrication and Omission

Run this on your own stack this week. Take the task your agents actually perform on regulated or contractual output. Build a held-out set with human-labeled ground truth, then stop reporting one accuracy number and report two rates instead.

Fabrication rate: fields the model populated that the source does not support. Omission rate: fields required by the schema that the model left empty when the source did support them. Both are deterministic counts against labeled truth, and neither requires an LLM to compute.

The ClinReg validator shows the shape of a scoring rule that resists gaming. It starts at 10 and subtracts 1.0 per error, 0.3 per warning, and 0.1 per information note, each capped, with a floor of 1.0. Invariants are asserted rather than inferred: exactly 254 subjects, and AVAL = BASE + CHG. On the cell-match task, per-file match rates are weighted equally at plus or minus 0.01 tolerance, specifically so a model “can’t inflate its score by nailing the two huge listings (roughly 88k and 12k cells) while missing the 28-cell primary efficacy table.” Equal weighting per artifact defeats the volume trick. Adopt that weighting or your dashboard will reward a model for being right about the parts nobody reads.

Once you have both rates, the selection decision changes shape. You are no longer asking which model is best. You are asking which failure your downstream process can absorb. A pipeline with a mandatory human completeness check tolerates omission cheaply and cannot detect fabrication at all. Pick accordingly.

Check Two: Calibrate the Judge Before You Trust It

Most teams that grade agent output at scale use a model as judge. ClinReg measured its judges before using them, and the measurements are large enough to invalidate an uncalibrated setup.

On a 0-10 scale, Gemini 3.1 Pro and GLM 5.2 came out lenient, averaging 8/10. GPT 5.5 came out strict, averaging 6/10. Two full points of systematic offset sit between judges on identical work. Self-preference deltas ran the same magnitude: plus 1.23 for Gemini 3.1 Pro and plus 0.67 for GLM 5.2, against minus 0.14 for Opus 4.8. A team using a single judge from the same family as the model under test is reading a bias, not a score.

Their structural answer is worth copying whole. Judges emit source-verified evidence rather than scores. Only findings confirmed by at least three of four judges count as correct. A separate “Chairman” model then assigns the score from the confirmed evidence. Their own summary of it: “Don’t trust a single LLM judge.”

The runnable version for your stack:

Grade 30 to 50 human-labeled items with each candidate judge and record the mean offset per judge. That offset is your calibration constant.
Run each judge against output from its own family and from other families, then subtract to get the self-preference delta. Anything above half a point disqualifies that judge for that family.
Change the judge’s job from scoring to citing. Ask for the span of source text that supports or contradicts a claim.
Require agreement from a majority of independent judges before a finding counts, and let a separate model do the arithmetic.

This costs more than one judge call, which is exactly the tension we mapped in the eval cost bottleneck. The cost is also the reason most teams skip calibration and then act on numbers that are off by two points in an unknown direction.

What the Benchmark Cannot Tell You

The authors disclose their own confounds, and the disclosures are the strongest evidence that the work is honest. Harness varies by model family: Claude Code for Claude, Codex for GPT, OpenCode for Gemini and open weights. Model and scaffold are therefore not separated. Their words: “The harness is part of the model. Every agentic result here is a model and a scaffold: validator strictness, stopping criteria, retry behavior. Under our setup, several results moved more with how a model was run than with which model it was.”

Read that as a warning about your own procurement. A benchmark rank you did not produce was measured inside a scaffold you do not run. It transfers poorly.

Two more caveats belong on the record. Fable 5 and Muse Spark 1.1 refused the benchmark tasks on biological-safety grounds and were dropped from the analysis, so the field is incomplete by construction. And Everest / Log10 sell evaluation services that this benchmark motivates, which makes the direction of the conclusion commercially convenient even where the method is sound. The per-model leaderboard was published only as an image; the figures above are the ones stated in prose.

One more finding is worth carrying into design. Task shape decides which behavior wins. On the extraction task, “restraint mattered more than persistence.” On the tables-and-listings task, “TLF rewards sustained iteration.” The same model tuned for one is wrong for the other, and no single leaderboard position captures that.

Do This Now

Pick your highest-consequence agent output and pull last month’s errors. Sort them into two piles: things the agent left out, and things the agent made up. If you cannot sort them because you never labeled ground truth, that is the first week of work. Then check whether whatever grades that output has ever been measured against human labels. A judge you have not calibrated is not a control, and a leaderboard rank you did not measure yourself is not a procurement justification.

This analysis synthesizes Open-Weight LLMs Have Caught Up on Accuracy (Suchismita Padhy, Wenzhe Xue, Niklas Nielsen and Arjun Bansal, Everest / Log10, July 2026).

Victorino Group helps regulated organizations build the measurement layer that makes model selection defensible to an auditor. Let’s talk.