The best score on Surge AI’s DAYJOB: Finance leaderboard is 23.9%, held by Claude Opus 5.5 (Adaptive/Max). Other entries include GPT 6 Astra (Max reasoning) at 21.5% and Claude Fable 5.1 at 19.8%. The page labels the column only as ”%”. It does not say whether that is a share of tasks passed, a rubric score, or something else, so I will not call it a pass rate or an accuracy figure. What the number does tell you is ordinal: on this set of finance work, the models I cite sit between 19.8% and 23.9% on whatever scale Surge chose.
The worked examples on the same page are more useful than the leaderboard. They show how the agents fail, and the two I quote here share a shape. The agent receives numbers, builds on them without checking them, and hands back a document that reads as finished.
A green light built on flipped signs
The page labels one example “Grok 4.6 green-lights an expansion on sign-flipped sales forecasts.” The agent recommended “proceed with Nashville only,” supported by a “pro forma Debt/EBITDA ~2.66x vs. 3.00x covenant.” On the agent’s own arithmetic, that leaves roughly a third of a turn of headroom under the covenant. A reader skimming the memo sees a ratio, a limit, and a recommendation that respects the limit.
The rubric expected something else. It expected the agent to notice sign errors in the sales forecasts, correct them, and produce a ratio “within [3.35x, 4.95x].” Even the bottom of that band sits above the 3.00x covenant. The correct conclusion is a covenant breach, so the right answer and the delivered one disagree on the only question the memo exists to answer.
In the lines the page quotes, the ratio is compared to the covenant and the recommendation follows from that comparison. A reviewer who checks only that chain finds it consistent. The defect sits in the inputs, one layer below the chain.
A release sign-off that missed the unit error
A second example reads “GPT 5.6 Sol misses the $25 million cents-versus-rand error.” The agent’s output included the line “No other release blocker was identified.” That sentence is a sign-off. It tells whoever reads it that the review is complete and the document can move.
I read the two examples together because the tone is the same in both. The quoted lines do not hedge, and they flag no doubt about the source data. The register is confident whether or not the numbers underneath deserve it.
A word of disclosure on the source. The DAYJOB page has no byline and no date, and I date it to September 2026 from the models it lists. Surge sells RL environments and data, so a benchmark showing that frontier models struggle with finance work is also a description of a market Surge serves. None of that makes the worked examples false. It does mean the leaderboard figure deserves less weight than the worked examples, which you can judge on their own terms.
Where Deel puts the arithmetic
Forkast published an examination of Deel’s back-office agents, and one passage describes a design choice that speaks directly to the Nashville memo. According to Deel, for tasks requiring precision, such as calculating tax rates, verifying account numbers and matching payment references, “the system uses deterministic rules rather than model judgment. AI reasoning is reserved strictly for genuine judgment calls.” Deel also says that “any high-stakes decisions are routed to human operators via Slack or email.”
Every figure in that piece comes from Deel. Forkast says so plainly: “all financial and operational metrics cited here are Deel-sourced and self-reported. There is no independent third-party audit available to verify the $140 million ARR figure.” So I am not citing Deel for its results. I am citing it for where it draws the line. Deel’s list names tax rates, account numbers and payment references. Signs and units are not on it, but my reading is that they belong to the same class: facts with one right answer, which a rule can check.
A second-hand report pointing the same way
The SAP LeanIX engineering blog carries a similar account, and it needs a heavier hedge. The story comes second-hand from a video by a contracting firm the post does not name, and its date is inferred. The post, credited to Jernej Klancic, retells the story of a pharma revenue-anomaly agent: “Initially, every step ran through an LLM. The results were unreliable.” The post’s own title is “Don’t go all in on AI.”
I would not build an argument on this account alone. Next to the DAYJOB examples and Deel’s stated design, it points the same way. Taken together, my reading is that precision work does not belong in the model’s reasoning.
Review the layer below the memo
We have written before about repeatability across runs and about the deterministic code around the model as a control. This essay concerns a different layer. The Nashville memo could be perfectly repeatable and still wrong every time, because the flipped signs arrive before the model starts reasoning.
My reading of the benchmark examples is that a review which reads only the memo is pointed at the wrong artifact. Picture a reviewer who receives the memo, reads the recommendation, checks that the ratio compares correctly to the covenant, and approves. That review would have passed the Nashville memo. It would have passed “No other release blocker was identified.” The check that would have caught both is upstream: are the forecasts signed correctly, and is this figure in cents or rand?
In the lines the page quotes, that check does not appear.
Do this now
Pick one finance workflow where an agent already writes a memo, a sign-off, or a recommendation. Then make four changes before its next run.
First, list every input figure the agent consumes and write down the invariant each one must satisfy: expected sign, expected unit, expected range. A sales forecast has a sign. A currency amount has a unit. Those invariants belong in code that runs before the agent sees the data, and a failed check should stop the run instead of passing a warning into the prompt.
Second, require the memo to restate the inputs it used, with their units, next to the conclusion. A reviewer who sees a line such as “sales forecast: sign checked” (my example) next to an expansion recommendation has something to question. A reviewer who sees only “~2.66x vs. 3.00x covenant” does not.
Third, change what reviewers sign. The approval should cover the inputs as well as the recommendation. If your reviewer checklist has no line about source data, the agent’s input handling is unreviewed by design.
Fourth, route the decisions that matter to a person, as Deel says it does for high-stakes calls. A covenant test is high-stakes by definition.
None of this requires a better model. The top score on DAYJOB: Finance is 23.9% on an undefined scale, and the two failures quoted here are ones a deterministic check on the inputs is built to catch.
This analysis synthesizes DAYJOB: Finance (Surge AI, September 2026), Deel Built 10,000 Agents for Its Own Back Office, Then Added $140M in Revenue Without Hiring. The Self-Reported Math, Examined (Forkast, September 2026), and Don’t go all in on AI (SAP LeanIX Engineering, September 2026).
Victorino Group helps finance and operations teams put deterministic input checks and reviewable sign-offs around their agents. Let’s talk.
All articles on The Thinking Wire are written with the assistance of Anthropic's Opus LLM. Each piece goes through multi-agent research to verify facts and surface contradictions, followed by human review and approval before publication. If you find any inaccurate information or wish to contact our editorial team, please reach out at editorial@victorinollc.com . About The Thinking Wire →
If this resonates, let's talk
We help companies implement AI without losing control.
Schedule a Conversation