Correct Once Is Not Correct: The Consistency Metric That Dropped Accounting Agents to 2.6%

Run the same accounting task eight times, with the same model, under identical conditions. Count how often the model gets it fully right all eight times. On APEX-Accounting, the benchmark Mercor built with Ramp across 160 tasks and nine frontier models, the most consistent model cleared that bar on 2.6% of tasks.

The same benchmark reports that at least one model earned some credit on more than 95% of tasks. Both numbers are true, and the distance between them is the whole story. Almost everything is solvable by something, once. Almost nothing is solvable by anything, reliably.

That distinction has no consequence for a chatbot. It has an enormous consequence for a process that writes to a record of account.

What Eight Runs Reveal That One Run Hides

Leaderboards report means. On mean criteria passed, APEX-Accounting looks like a normal frontier benchmark: Claude Fable 5 at 56.4%, Muse Spark 1.1 at 52.6%, GPT-5.6 Sol at 51.5%, down to Qwen 3.5 at 24.4%, the lowest of the nine. Read that column and you would conclude the top models are somewhere past halfway to automating month-end work, with a clear ordering and a plausible improvement curve.

Now change the aggregation. Pass@8, meaning the task is solved on every one of eight attempts, puts Muse Spark 1.1 at 21.5% and Fable 5 at 20.1%. Require full correctness on all eight runs and the leader falls to 2.6%. And 58% of the 160 tasks were never fully solved by any of the nine models on any run.

The mean is not wrong. It answers a question that does not govern anything. A mean of 56% describes an expected value across a population of attempts, which is useful when you are shipping a hundred low-stakes drafts and a reviewer skims them. It tells you almost nothing about what happens on the one attempt you actually put into the ledger. For that, the number you need is how often the same input produces the same correct output.

Accounting is where this bites first, because accounting has an unusual property: the artifact persists, it is signed, and it is compared to the prior period. A marketing agent that varies across runs produces variety. A close agent that varies across runs produces reconciliation work, and the reconciliation is more expensive than the original task.

Seven in Ten Failures Are Judgment, Not Lookup

The instinct when a model fails on enterprise data is to blame retrieval. Wrong document, missing context, bad chunking, stale index. That instinct is measurable, and on this benchmark it is mostly wrong.

Roughly seven in ten graded failures traced to flawed reasoning rather than retrieval. Fable 5: 19 of 24 graded failures. GPT-5.6 Sol: 21 of 28. Muse Spark 1.1: 13 of 22. The models found the right numbers and then did the wrong thing with them.

The benchmark authors put the operational consequence plainly: “A model that drops a correct intermediate answer can still create a bad journal entry.” That is the failure mode that survives every RAG improvement you can buy. Better retrieval raises the ceiling on what the model can see. It does not touch what the model concludes.

The same report draws a line that matters for anyone using exam-style evidence to justify deployment. “This is a very different result from performance on accounting exams. Models can produce the right answer in a controlled test and still fail to carry accounting judgment through a real workflow.” An exam scores a terminal answer. A workflow requires carrying a judgment across a dozen dependent steps without dropping it once.

The method behind these numbers is heavier than most benchmark work. 160 tasks across 10 simulated companies. More than 40 accounting professionals with a median 11 years of experience, over half of them ex-Big Four, writing rubrics that average 13.7 criteria per task. An open-source AI judge that reaches 97% agreement with the expert graders. This is not a scraped question bank.

Budget Caps Behave Differently Per Model

The cost finding is the one most likely to be misread, and it inverts a common procurement assumption.

Fable 5 scores 11.8% under a one-dollar per-task cap and 55.2% under a fifty-dollar cap. That is a real capability unlock from budget alone, on the same model. The report states percentages only at those two endpoints, so any figure someone quotes for a five or ten dollar cap is read off a chart, not off the paper.

The interesting part is what happens at the top of that range. At the fifty-dollar cap, Fable 5 spends around 32 dollars per run. Muse Spark 1.1 spends around 5 dollars and lands within four percentage points. Across the board, models use only 64.7% of the maximum budget available at that cap, meaning the ceiling is not what binds them.

A single global spend cap therefore does something different to each model in your stack. Set it at one dollar and you have quietly disabled the model you selected for its ceiling. Set it at fifty and you are paying six times more for a four-point difference on one of them. Budget policy has to be written per model and per task class, tuned against the actual spend curve, not applied as one org-wide number in a gateway config. We have written before about agents that ignore the budget they were given; this is the mirror problem, where the budget silently rewrites the capability you thought you bought.

What the Benchmark Does Not Cover

The exclusions are broad enough that they change how far the result travels. APEX-Accounting explicitly leaves out tax, audit, consolidation, multi-entity and multi-currency accounting, and external reporting. It also excludes clarification-seeking behavior, which means a model that would have asked a sensible question in real life gets no credit for that instinct here.

Those are among the hardest parts of a real close. The 2.6% consistency figure comes from the simpler half of the domain.

There is a second caveat worth carrying. The companion Ramp Harness result is described by its own authors as a harness ablation, not an evaluation of the complete Ramp Stack product. Treat it as a study of scaffolding effects, and do not let it circulate internally as a product benchmark.

Repeatability Is the Number to Govern

The governance question for any agent that writes to a record of account is not what score it averages. It is how many identical runs produce identical, correct output, and what happens on the runs that do not.

That reframes several things at once. Acceptance testing for a finance agent should be N-run, not single-run, with N chosen so the variance is visible rather than flattered. Vendor claims quoting mean scores should be treated as incomplete until the consistency distribution is disclosed. And the human review step cannot be sized off the mean either, because a process correct 56% of the time on average and consistently correct 2.6% of the time needs review on essentially every artifact, not on a sample.

This is the operational texture the governance-beyond-engineering arc keeps surfacing. When vendor templates draft regulated artifacts, the signoff perimeter extends to the template. When those templates are non-deterministic at the rate this benchmark measures, the perimeter extends to every single run.

Do This Now

Pick the one accounting or finance workflow you are closest to automating. Take five real tasks from it, run each one eight times through your current model and prompt, and grade all forty outputs against your own criteria. You now have a consistency rate for your own stack, which no leaderboard can give you.

Then set your review policy from that rate rather than from a benchmark mean. If consistency is under 25%, the agent is a drafter with mandatory line-level human review, and staffing has to reflect that. If a task class never lands in the consistent bucket across eight runs, it stays manual until it does.

Last, measure your spend curve per model before you write a budget cap. Run the same task set at two or three cap levels and look at where each model’s score stops moving. The number that maximizes one model’s capability may be six times more than another model needs for nearly the same result.

Nine models, eight runs, 160 tasks, and no frontier model can close a month yet. That is not a reason to wait. It is the specification for what to instrument while you build.

---

This analysis synthesizes Introducing the AI Productivity Index for Accounting (Mercor, in collaboration with Ramp, July 2026) and Ramp SWE-Bench (Ramp Labs, July 2026).

Victorino Group helps finance and operations leaders design N-run acceptance tests and per-model budget policy before an agent touches the ledger. Let’s talk.