Chrome Calls Its Decision Scores Calibrated. The Code Divides by the Sum

“The interface comments call the output calibrated. The code does no calibration step beyond the division above.” Dan Petrovic wrote that sentence after reading the code reviews behind Chrome’s experimental DecisionModel interface. The division he refers to is short enough to quote whole. Chrome’s probabilities are “The engine’s raw scores, with negatives set to 0, divided by their sum”.

That is a normalization. It makes the numbers add up to one. It says nothing about whether an option scored at 90% turns out right about nine times in ten.

The distinction matters because of what Microsoft pitches decision models to do: sit on the control path of an agent, where applications “use confidence to decide when to act, defer, or ask for review”. In practice that means thresholds: act above one number, defer between two, escalate below. Microsoft states the requirement in its Decision-1 launch post, in one sentence: “Applications use confidence to decide when to act, defer, or ask for review, so a 90% prediction should be right about nine times out of 10 on representative cases.”

Hold Chrome’s code against that definition and the word “calibrated” in the interface comments becomes a claim the code does not back.

What Petrovic found in the reviews

Petrovic’s write-up on DEJAN is concrete because the code reviews are open. He quotes them directly, and two of the problems the reviewers found are the kind a threshold alone does not reveal.

The first was an early bug, caught against a reference, where “later options matched the reference 0/27”. The second is position bias. On Gemma 4 E2B the reviews report a small effect, 1.3 points or less, and on TinyGemma a severe one: it “picked A in 440 of 480 cases”. In principle, a model that lands on the first option most of the time could still emit scores that look confident, and after the division they would still sum to one. Nothing in the arithmetic flags that the confidence is tracking position rather than correctness.

Both figures come from open code reviews of a prototype still in review. Chrome’s engine code “is still in review and may change before release”, and the bias numbers describe a development snapshot, not a shipped product. They are useful for a different reason. They show what turned up only because reviewers measured, and a team building on a decision API learns nothing about its scores until it runs the same kind of test.

The 0.85 problem

Petrovic also ran a test of his own, reproducing Chrome’s EmbeddingGemma engine outside Chrome. No answer in Chrome’s format reached 0.85, the threshold Google’s explainer uses before applying a filter.

Consider what that means for a developer following the explainer. They set the filter at 0.85 because the documentation suggests it. In Petrovic’s ten-question test, no answer in Chrome’s format got that high, so a filter set there would not have fired once, and the developer would see a quiet system that looks like it is working. A threshold borrowed from documentation encodes someone else’s score distribution, and when the distribution moves, the threshold silently stops meaning anything.

Petrovic is explicit about the limits of his test: “Our test uses 10 questions written and marked by one person, so it shows the direction of the differences between engines, not their size.” That caveat deserves to travel with the finding. Ten questions cannot tell you how far off Chrome’s scores are. They can tell you the 0.85 threshold and the scores of Chrome’s format, as he reproduced it outside Chrome, do not obviously belong to the same scale, which is enough to justify measuring before relying on either.

Microsoft’s definition, and Microsoft’s own table

Microsoft’s launch post is secondary evidence here, for a reason we have to disclose. Our editorial pipeline scores articles with Jev, built by TypeSafe, and Microsoft benchmarks Decision-1 against Jev. Read what follows with that in mind. It is also why this piece leads with an independent code review of Chrome and uses Microsoft mainly for its definition.

The definition is the useful part. “A 90% prediction should be right about nine times out of 10 on representative cases” is testable. It names the property, it names a ratio, and it names where to measure: representative cases. That last phrase carries most of the weight. Calibration measured on a vendor’s benchmark tells you about the benchmark.

Microsoft’s own post then reports its standing on that property. Microsoft-Decision-1 is “Third on calibration”, at 92.2, behind Jev 1.13.0 at 93.7 and Quyet-1.0-Large at 93.1. These are self-benchmarks, published by the vendor in its launch post. We cite them for one narrow point: a vendor that defines calibration carefully also reports it as a separate, ranked measurement. Calibration is a property each model earns or fails to earn. Calling a model a decision model does not confer it.

The same post shows why benchmark tables need reading line by line. Decision-1’s latency was “measured through Foundry in the same region”, while the other models’ latencies come from JevBench medians. Two measurement setups share one comparison. That does not make the latency figures wrong. It makes them figures from different rooms, and a reader who skims the table will compare them as if they came from one.

Why the word is doing so much work

A raw score and a calibrated probability look identical on a dashboard. Both are numbers between zero and one. Both can be thresholded. The difference only appears in aggregate, when you group predictions by their stated confidence and count how often each group was right.

A team can wire a threshold on the strength of the label. If an API says “calibrated”, a reasonable developer reads a 90% score as a 90% chance and sets the act threshold there. If the number is a normalized raw score, the hit rate behind it on their data is unknown until someone counts. It could be far lower or higher. The threshold then encodes a policy nobody chose.

We have argued before that a gate should be judged by how it fails, and that three decision models leave the decision to abstain to the caller. The first post already proposed testing confidence before setting a threshold. This piece adds what the code shows. Calibration sits underneath both arguments. A defer band between two thresholds only works if the numbers inside the band mean what the band assumes. An abstain path driven by uncalibrated confidence can abstain on the wrong cases. And when a check misbehaves, the place to look is the check itself, which here means the step that turns scores into probabilities.

Test before you threshold

This recommendation is ours. Neither source makes it. It rests on Microsoft’s definition, applied to whatever model you are about to wire into an agent.

Before any confidence number becomes a policy threshold, run a reliability check on your own representative cases:

1. Collect labelled cases from your real traffic. Sample the decisions the agent will actually face, with the correct answer recorded by someone who knows the domain. Petrovic’s caveat applies to you as well: a small set written by one person shows direction, not size. Size the sample to the decision.
2. Bucket predictions by stated confidence. Use narrow, fixed bands of stated confidence and put every prediction in its band.
3. Count hits per bucket. If the bucket around 90% is right far less often than about nine times in ten, the score is not calibrated on your data, whatever the interface comment says.
4. Check the score distribution against the threshold you plan to use. If no output ever reaches your threshold, as Petrovic found with 0.85 in his reproduction, the gate is decorative.
5. Look for position bias. Shuffle option order and rerun. A model that picks the first option in most cases, as TinyGemma did in the reviews, can pass a naive accuracy check on one dataset and fail on the next.
6. Read the normalization step, if you can. In Chrome’s case the code and its reviews are open, and they show raw scores divided by their sum with no calibration step after it. Where the code is closed, ask the vendor what step produces the probability and on which data it was fitted.

Then set thresholds from what the buckets show, record the date and the model version, and repeat the check when either changes. Petrovic notes the engine code is still in review and may change before release. A threshold tuned to today’s normalization can drift with the next engine.

The word “calibrated” in an interface comment is a statement of intent. The bucket counts on your own cases are the evidence. Only the second one should set policy.

---

This analysis synthesizes Chrome Decisions API: the prompts, limits, engines and code reviews behind DecisionModel (DEJAN, October 2026) and Introducing Microsoft-Decision-1, our model for fast decision-making (Microsoft, October 2026). Disclosure: Victorino Group scores articles in its editorial pipeline with Jev, a model Microsoft benchmarks against.

Victorino Group helps engineering teams test the confidence scores behind agent decisions before they become policy. Let’s talk.