0.000 Accuracy at 0.952 Confidence: Move the Escalation Trigger Off the Model

Khmer: 0.000 accuracy at 0.952 mean confidence. That line comes from the evaluation page ConvAI Innovations published for Laya, a multilingual decision engine, and the page spells it out: “Not one correct decision in 100 questions, while reporting ~95% confidence.”

Other rows read the same way. Armenian scored 0.050 accuracy at 0.885 confidence. Hebrew scored 0.060 at 0.964. Bengali, 0.080 at 0.945. Hindi, 0.100 at 0.941. For scale, the page puts random chance on its 20-option MASSIVE sweep at 0.050.

The sentence that governs the whole table is this one: “Across 51 languages, the English checkpoint’s mean confidence never drops below 0.885, regardless of whether its accuracy is 82% or 0%.”

Take an escalation rule of the form “send the case to a human when the model’s confidence falls below 0.9”. Apply it to the averages above. Every row except Armenian sits above 0.9 on average, so the rule would leave most of those cases with the model. Armenian averages 0.885, just under the line, while its accuracy sits at 0.050. The rule was built for the moment the model does not know. That moment is where it goes quiet.

A correction to what we published in May

In May we argued that calibrated confidence is the missing governance primitive. That post told operators to define routing thresholds, escalation rules and review queues in terms of confidence, so that trusting the number would become a configuration change once models were calibrated.

Part of that needs correcting, and the correction belongs up front. Calibration makes a confidence number honest about what the model knows. It says nothing about what the model cannot see at all.

The Laya page makes the distinction measurable. It reports that expected calibration error falls from 0.466 to 0.081 by fitting a single scalar temperature per question type on the target domain distribution. That is a large repair, and it is cheap. The confidence-without-competence failure in the table persists after it. My reading of why: a model that cannot read Khmer script has no internal signal telling it so. Temperature scaling rescales a signal. It cannot create one that was never there.

The page draws the conclusion in bold, and I quote it: “the model’s own confidence gives no warning when it cannot read the input script”, and “Therefore, confidence gating cannot protect you. The decision of which model to use must be made before the forward pass.”

So calibration stays necessary. What changes is where the escalation trigger lives. Any rule that asks one model how sure it is inherits that model’s blindness to inputs it was never built for.

Half one: decide routing before inference

The first half of the fix is the Laya page’s own conclusion, applied to escalation. If the model cannot warn you, the decision cannot wait for the model. It has to be made from something you can check deterministically, before the forward pass, about the input itself.

The worked example here is mine, not the page’s. Take a support-ticket classifier trained and evaluated on English. Before any ticket reaches it, a pre-check identifies the script the ticket is written in. Latin script goes to the classifier. Anything else goes to a model evaluated on that script, or to a human queue, and the classifier never sees it. The pre-check has no opinion about the content. It answers one question the model cannot answer about itself: is this input inside the population the model was measured on?

That question should feel familiar. We have written about thresholds measured on a population that is not yours, and about who signs off on a classifier threshold. A pre-check is the operational form of the same concern. It keeps a model’s numbers attached to the inputs those numbers were measured on.

What goes into the pre-check depends on your system. Script, language, schema version and file type are candidates. So is whether a field the model depends on is present at all. The common property is that each check runs outside the model and returns a yes or a no that does not depend on the model’s opinion.

Half two: escalate on disagreement between independent judges

The second half is already running in production, at Cloudflare.

Cloudflare published a write-up this month on how its client-side security protects storefronts from malicious scripts. The escalation rule is one sentence: humans review only what is flagged malicious “or lacks a clear two-thirds majority”. The vote comes from an ensemble drawing on “leading models from around six different families”, each run as an agent in a fresh independent session, with votes weighted by the model’s score in the Artificial Analysis Intelligence Index.

Then comes the sentence I would put on a slide for any team designing an escalation path: “The frontier models sometimes disagree, especially on the most intricate scripts. We treat that disagreement as signal, not noise.”

Read it against the Laya table. A single model reporting 0.952 on input it cannot read gives you one number and no way to doubt it. Models from around six families, each in its own session, give you a distribution. When they split, the split is the warning. No model has to know it is wrong. It only has to fail differently from the others.

Cloudflare also shows the funnel in front of human review. Under 0.3% of all analysed traffic is flagged malicious by the first-stage graph neural network, and only those go to a second-opinion model before a human is involved. Human attention goes where a flag or a split vote exists.

Cloudflare’s case for its detection stack rests on a measured blind spot in incumbent tooling. Across “four operations, spanning eight payloads”, Cloudflare reports that “seven of the eight payloads were entirely absent from VirusTotal, and URLScan returned no malicious verdict for any of them. Page Shield ML, meanwhile, caught all eight in live traffic.” Absent from VirusTotal means VirusTotal had no record of them. It does not mean VirusTotal called them benign. Cloudflare holds its own claims to the same standard. For the eighth payload it says “public history does not reveal when that verdict was first assigned”, and whether any request produced completed attribution or paid commission “remains unobserved”.

Read both sources as vendors

Both pages are vendors arguing for their own products, and the figures should be held to the conditions each page states.

Every Laya number is measured by the vendor. The page says so itself: “Every Laya number is measured; Jev numbers are published by third-party independent studies (AbdelStark, nibzard) and TypeSafe AI.” The page also reports a counter-case against itself: on Banking77, with 77 labels, Laya scores 0.425 against 0.870 for Jev, the comparison system the page benchmarks against. I use none of the page’s speed claims. The only Laya material this argument needs is the confidence-without-competence table.

Cloudflare’s eight payloads come from four operations. That is a small sample chosen by the company that caught them. It shows that Page Shield ML caught those cases where VirusTotal and URLScan did not. It does not establish a detection rate, and the page does not attribute the catch to the ensemble vote specifically.

One more boundary. The two sources are independent, and neither was written in response to the other. The convergence is my synthesis. Laya shows why a single model’s confidence cannot be the trigger. Cloudflare shows a trigger that does not depend on any single model’s confidence. Neither page makes the other’s argument.

The two-part rule

Put together, the rule for any escalation path is short.

Route before inference. Decide which model, if any, sees an input from a deterministic check on the input itself. A model’s numbers only apply to the population it was measured on, and the check enforces that.
Escalate on disagreement. Among inputs that pass the check, send to a human what independent judges split on, plus what they flag. One model’s self-reported confidence can feed the vote. It cannot be the trigger.

Calibration still has a job inside that structure. A calibrated confidence makes each vote more honest and makes weighting the votes meaningful. Our May post was right that it is a primitive. It was wrong to let it sit alone at the gate. For LLM judges specifically, our note on Netflix’s golden-set methodology covers how to evaluate each one before it votes.

Do this now

Pull up every escalation rule your AI systems run in production and sort them into two lists. The first holds rules that fire on a single model’s confidence score. The second holds rules that fire on something outside the model: an input check, a second model, a vote.

For each rule on the first list, answer one question in writing: what happens when the input is something the model cannot read? If the honest answer is “the model reports high confidence and the rule stays silent”, you have a Khmer row in your system. Add a pre-check in front of the model that tests the input against the population it was evaluated on. Then pick the highest-risk decision on the list and give it a second, independent judge from a different model family, with disagreement routed to a human.

Start with the inventory and one pre-check. The vote can follow.

This analysis synthesizes Laya: A Multilingual System 1 Decision Engine (ConvAI Innovations, September 2026) and When scanners miss the attack: how Cloudflare client-side security protects storefronts (Cloudflare, September 2026).

Victorino Group helps engineering teams redesign escalation paths so the trigger does not depend on one model’s opinion of itself. Let’s talk.