Your Agent Averages 77.4% and Succeeds Every Time on 53% of Tasks

IBM Research ran ReAct with GPT-4.1 five times over the 168 tasks of AppWorld’s test_normal split. Mean@5, the average success rate, came back at 77.4%. Pass^5, the share of tasks the agent solved on all five runs, came back at 53.0%. The agent temperature was 0.0.

So 24.4 points of that headline score belong to tasks the agent sometimes finishes and sometimes does not. No determinism checklist touches that number, because the temperature was already pinned. On harder tasks the distance is worse: IBM’s write-up says it “reaches 30 points.”

This is the number that decides whether an agent can be operated, and we almost never see it reported.

The Gap Between Benchmark Averages Is Smaller Than One Configuration’s Variance

We introduced this measurement in Evaluation-Driven Development, using ServiceNow’s EVA framework, where the delta between pass@3 and pass^3 was substantial across every system tested. The conclusion there was that capability and reliability are separate metrics needing separate infrastructure. IBM’s result is what that separation costs when you put a number on it at temperature zero.

Dan Luu, writing in 2026, arrived at the same place from the opposite direction and made it worse. He measured one standard deviation between runs of a single configuration at 0.075, or 7.5%. Then he measured the spread between the best and the worst configuration he tested across all the GPTs: 1.055 against 0.986. He describes that spread as less than one standard deviation. The distance between those configurations is inside the run-to-run noise of a single configuration.

The consequence for rankings is immediate. On DeepSWE, with roughly 100 tasks run four times, changing a single task flips the top two models. A leaderboard built on that data is not reporting which model is better. It is reporting which run happened to get sampled.

The same trap shows up at smaller scale. A setup labelled caveman mode looked like a large win at n=2, $12.45 against $40.38. At n=50 it was 1.03 against 1.01 speedup. Noise, measured twice, reads as a finding.

A Bigger Model Is the Wrong First Move

In our experience, procurement treats consistency as something capability buys for you. IBM’s page does not support that. Its claim is narrower: a stronger model raises Mean@k, and it does not necessarily narrow the distance between Mean@k and Pass^k. Those are different axes. Paying for the first one does not entitle you to the second.

What did move the second axis did not require a new model. Guidelines mined from a single baseline trajectory took Pass^5 from 53.0 to 69.0 and Mean@5 from 77.4 to 81.0. The 24.4-point consistency gap closed to 12.0. Mean@k never dropping was a stated hard design requirement, not a lucky outcome, and that matters, because in our experience a reliability intervention that costs peak capability is the one that gets rejected.

The lift concentrates where operations hurt. Pass^5 rose 22.9 points on Medium tasks, a 44% relative improvement, and 14.3 points on Hard, 45% relative. Easy tasks gained 12.2 points.

We covered the original guideline-mining work in the governance question nobody was answering, where the objection was that scoring a guideline by task completion validates the outcome without validating the rule itself. That objection stands. What has changed is the evidence about what the rules contain. Guidelines transferred to similar tasks gave +13.0 points, only 3 points below the +16.0 on the same task. On the weaker gpt-oss-120b the ordering inverts: the similar-task lift of +8.7 points exceeds the same-task lift of +6.0 (10.1% to 16.1%). A memorised trajectory cannot generalise better off-task than on it. That is the page’s own evidence that what got extracted is a transferable rule.

The Detection Runs on a Trace You Already Have

The operational property matters more than the benchmark delta. Detection is black-box: no logits, no ground truth, no end-to-end replay. It costs one extra model call per decision step, at k=5, and the mining ran from one already-recorded baseline trajectory.

That is the only shape production oversight can afford. Replaying an agent end to end against ground truth is a research budget. Reading a trace your observability stack already wrote, and paying one extra call per decision, is a line item.

The Only Provenance Number Here Is Hand-Coded

One more signal, and it needs its bias stated before its number. A student who explicitly discloses hating LLMs hand-coded the F-Droid update batch of 2026-09-12 and classed 74 of 102 apps, 72.5%, as largely AI-written. One survey, one coder, one declared prior. Treat it as directional and nothing more, and never as a measurement of code quality, because it measures who wrote the code and not whether the code works.

Its disclosure discipline still exceeds most published AI-adoption surveys, which is the uncomfortable part.

Meanwhile the failures in Luu’s record are the consistency problem wearing different clothes. Codex produced a convincing Playwright video of a bug repro that turned out to be, in his words, “an artificial browser environment that was designed to create a fake repro, not the real environment.” Opus 4.8 reported 514% of resources consumed on a task where anything above 100% is impossible. Running 10 concurrent agents, he puts the rule-violation rate at about one per 100 rule-agent-days. An agent that is right on average produces artifacts that look right on average.

Do This Now

Add one column to your agent evaluation and one gate to your model selection.

The column. Every agent report that carries an average carries Pass^k next to it, at k=5, at the temperature you actually ship. If your current sign-off document has a single accuracy figure on it, you do not know your agent’s consistency at all, and the number you are missing was 24.4 points wide in a controlled setting.

The gate. A benchmark delta stops counting as procurement evidence unless it exceeds the run-to-run standard deviation of a single configuration. Luu’s 7.5% is a published starting point. Measure your own by running one configuration repeatedly on your own tasks, which in our experience costs less than the model migration the leaderboard is recommending.

Then, before approving a spend on a larger model to fix reliability, mine guidelines from one recorded trace of the agent you already run. In IBM’s setting that halved the consistency deficit without costing average performance. It is a smaller commitment than a migration.

In the data here, the differences between the benchmark averages used to choose models are smaller than the run-to-run variance inside a single configuration. Once you report both numbers, the second one starts driving the decisions, which is the correct outcome. Consistency is what the customer experiences. The average is what the vendor sells.

---

This analysis synthesizes IBM Research’s Your Agent Aced the Task. Will It Do It Again? (IBM Research, September 2026) and Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course (IBM Research, September 2026), Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island (Dan Luu, 2026), and How much of F-Droid is LLM generated? (tintotint, 2026).

Victorino Group builds the consistency measurement and sign-off gates that let enterprises put agents into production without trusting an average. Let’s talk.