Your Benchmark Score Is an Average Over a Cliff: Surprisal-Graded Eval

On one KramaBench question, an agent scored F1 1.00. Delete a single high-information term from the query, the token “TLE”, and the same agent scored 0.00 on the same question. The question did not get harder in any way a human would notice. One word carried almost all of the retrievable signal, and removing it collapsed the answer.

That collapse is invisible in the number your benchmark reports. A pass/fail suite runs each query once, at one phrasing, and averages the results. The average sits on top of a curve that can fall off a cliff a few words away, and the report never shows you the cliff. Google Data Cloud’s Frontier AI team published a method for drawing it, and the method is worth stealing regardless of what you think of their benchmark.

The method: grade every query by surprisal

The technique is called iSQR, and the idea is small enough to reimplement in an afternoon. For each query in an eval set, you measure the information content of its terms using TF-IDF, the same weighting that ranked documents in search engines for two decades. High-TF-IDF terms are rare and discriminating. Low-TF-IDF terms are common and carry little signal. “TLE” in a corpus of astronomy questions is high surprisal. “the” is not.

Once you have a surprisal score per term, you generate calibrated variants of each query:

Easy: inject or preserve the high-information terms, so the query points straight at the answer.
Neutral: the original phrasing.
Hard: strip or paraphrase the high-surprisal terms, so the agent has to recover the intent from context.

Now you do not have one score per question. You have a curve per question, and you plot the agent’s F1 across the difficulty axis. The KramaBench result is what falls out: for at least one question the curve is a step function, 1.00 at neutral, 0.00 once the load-bearing token is gone. A single average of those two points would report 0.50 and describe neither.

This is the part that changes how you read any leaderboard. A model that scores 0.72 on a suite might be flat and reliable across phrasings, or it might be a stack of cliffs that happens to average to 0.72. The scalar cannot tell you which, and the two agents behave nothing alike in production, where users phrase things however they phrase them.

The sweet spot is not the easiest question

The counterintuitive finding is where the eval signal actually lives. Across the graded variants, medium-ambiguity queries scored higher than both easy and neutral ones: F1 0.81 for medium against 0.78 easy and 0.76 neutral. The easiest phrasing did not produce the best score.

The reason is that the easy variants over-specify. When you inject every high-information term, you hand the agent a query so pointed that retrieval succeeds trivially, and you stop learning anything about the agent’s ability to disambiguate. The medium band is where the agent has to do real work and can still succeed. That is the band that discriminates between a strong agent and a lucky one. Build your eval set entirely out of unambiguous questions and you will measure retrieval plumbing, not reasoning.

Validate the ruler before you trust the agent

Here is the finding that should stop you from shipping any difficulty-graded eval on autopilot. The verdict depends on how you generate the ambiguity.

Grounding the ambiguity in TF-IDF, deriving hard variants from the corpus statistics, produced an agent F1 around 0.85. Generating the ambiguity with an LLM instead, asking a model to “make this query harder”, produced an F1 around 0.34 on the same agent. Same agent, same underlying questions, two difficulty-estimators, and the conclusion swings by half a point of F1. One of those rulers is measuring the agent. The other is measuring the quirks of whatever model rewrote the queries.

The lesson generalizes past this one paper. Any eval that generates its own hard cases is only as trustworthy as its difficulty-estimator, and that estimator is a component you have to validate independently. A TF-IDF ruler is inspectable: you can list the terms it scored high, check whether they are genuinely discriminating, and reproduce the score deterministically. An LLM difficulty-estimator is a second black box stacked on the one you were trying to measure. If you cannot explain why your eval called a query hard, your eval is not measuring difficulty. It is measuring a model you did not audit.

The ground truth was quietly broken

The method also surfaced defects in KramaBench itself that a scalar score would have hidden. The team found 124 sharded tables that exceeded the agent’s retrieval limits, questions whose answers physically could not be retrieved within the context budget the agent was given. They found questions pointed at the wrong target tables, where the labeled answer came from a table the question did not describe.

An agent that fails those questions is not wrong. The benchmark is. And a single averaged score folds those broken items into the same number as the legitimate failures, so the model that correctly refuses an unanswerable question scores identically to the model that hallucinates an answer to it. Quietly broken ground truth means quietly wrong conclusions, and you only find it when you inspect the per-question curve instead of the aggregate.

Do this now

You do not need Google’s benchmark to use the technique. Take one eval set you already trust and run this pass on it:

Score the terms. Compute TF-IDF over your eval corpus. For each question, rank its terms by surprisal. The top one or two terms are your load-bearing tokens.
Generate three variants per question. Easy (terms preserved), neutral (original), hard (top surprisal terms stripped or paraphrased). Keep the difficulty-estimator inspectable. If you use an LLM to paraphrase, log what it changed so you can audit it.
Plot the curve, not the average. For each question, chart F1 across the three variants. Flag every question where the curve is a step function. Those are your cliffs, and they are where the agent will fail in production the first time a user drops the magic word.
Audit the flagged items for broken ground truth. A 1.00-to-0.00 cliff is sometimes a real capability edge and sometimes a mislabeled question. Both are worth finding. Only one is the agent’s fault.

The output is a map of where your agent’s capability actually ends. A leaderboard number was never able to give you that.

This analysis synthesizes Frontier and Center: Who Evaluates the Evaluations? (Google Data Cloud Frontier AI, July 2026).

Victorino Group helps teams build evaluation harnesses that show where AI capability actually breaks. Let’s talk.