One Prompt, One Model, 4x the Cost: Governing Agent Spend by Variance

Netlify ran one identical prompt across 11 coding models, three runs each, and published the per-run numbers. Claude Opus 5 averaged 519 credits. The three runs behind that average were 253, 249 and 1,055. Same model, same prompt, and a 4x spread inside a single row of the table.

That number is the argument. Most agent cost controls in production today are a monthly ceiling and an alert threshold, both of which describe the mean. The mean was 519. Nothing in a spending cap tells you that one run in three cost four times what the other two did, or which run it was, or why.

Netlify’s own read of the pattern is unusually blunt for a vendor blog: “Opus does have a tendency to run off with excessive credit usage (compared to its typical baseline) more than other models. It does not guarantee a worse or better outcome, though.” Cost blew up. Quality did not move with it. Those two facts together are what make variance a governance object rather than a curiosity.

The Distribution Is Wider Than the Model Choice

Read the full Netlify table and the model-selection debate shrinks. Averages across the 11 models: Opus 5 at 519 credits, Sonnet 5 at 143, GPT 5.6 Sol at 141, Gemini 3.6 Flash at 103, Kimi K3 at 102, Gemini 3.1 Pro at 53, GPT 5.6 Terra at 39, DeepSeek V4 Pro at 37, GLM 5.2 at 27, Kimi K2.7 Code at 19, and DeepSeek V4 Flash at 2.4. The field spans roughly 200x on one prompt.

Two hundred times between the cheapest and most expensive option, and 4x inside the most expensive one. A team that agonizes over which model to standardize on, then runs it uninstrumented, has optimized the smaller of the two quantities.

Treat the caveats honestly. This is a single prompt, three runs per model, published by a vendor that announces an OpenRouter partnership in the same post. Three runs is not a distribution, it is a hint of one. The specific credit figures should not be copied into anyone’s forecast. What survives the caveats is the shape: per-run cost on agent work is heavy-tailed, and the tail sits inside a single model’s own behavior.

What a Cap Actually Controls

A monthly cap is a circuit breaker. It fires after the money is spent, it fires on the aggregate, and its only available response is to stop everything. Between zero and the ceiling it is blind.

Three things a cap cannot do:

Tell you that a routine PR review consumed many times its usual budget, because that run is invisible inside a monthly total.
Distinguish an expensive run that shipped a migration from an expensive run that looped on a malformed prompt and produced nothing.
Authorize more money for the one task where more money was the right call.

That last one is the expensive omission. Databricks puts it directly in their Smart Routing post: “Valuemaxxing cuts both ways: take the cheap model when it suffices, and be confident in spending more when the value justifies it.” A cap only knows how to say no. A cost control that can only reduce spending will, over a year, systematically underfund the small number of tasks where frontier capability was worth paying for.

Routing Is the Control Surface, Telemetry Is the Instrument

Databricks reports that Smart Routing in their Unity AI Gateway outperformed any single model at 65% of the cost per task of a leading model like Opus 5. They cite 35% savings on an internal benchmark and 56% on public coding benchmarks, with 30%+ as their stated conservative floor.

Those three numbers come from different experiments and should not be added, averaged, or quoted as one result. The internal benchmark is described as one that no labs have had access to, which means no one outside Databricks can check it. Read it as a vendor’s directional claim from a vendor selling the gateway.

The mechanism is more useful than the percentages, and the mechanism has a stated weak point. Databricks: “Opening prompts are rarely precise, since what a developer types first is a symptom or a rough intention rather than a specification, and our router reads that first message and commits.” A router that classifies on the opening message inherits the imprecision of that message. So they recommend deploying first where the task statement is machine-written: PR reviews, sub-agent launches, batch migrations, scheduled jobs. In those paths the scope is already explicit, so classification is close to free.

That deployment order is the part worth stealing. It is also the part most teams skip, because chat is the visible surface and machine-issued work is not.

For monitoring, Databricks names three metrics: breakdown of sessions by model, number of sessions completed end-to-end by the routed model, and dollar savings from routing. The second one carries the weight. A router that cuts the model bill by sending work to a cheap model that then hands back a half-finished task has moved cost into human time, where nobody is measuring it.

Four Numbers to Instrument

Cost per completed task is the right denominator, and we have argued that case separately. Routing as a runtime control has its own treatment here. What neither covers is the distribution around that denominator. Instrument these:

Per-run cost, retained. Not a monthly total, not a daily rollup. One row per agent run with the task class, the model, the cost and the outcome. Without this row nothing else on this list is computable.
P50 and P95 per task class. The median tells you what the work costs. The 95th percentile tells you what your worst weeks look like. If P95 sits at 4x P50 on a class you run hourly, your forecast is wrong in a way no average will reveal.
Cost-to-outcome correlation. Take the top decile of runs by cost. What fraction shipped? Netlify’s finding was that excess credits did not guarantee a better result. If your expensive tail has the same completion rate as your median, the tail is waste and you can cut it. If it has a higher completion rate, the tail is your hardest work and capping it is the mistake.
Cap hits by task class. Every time a run terminates on a limit rather than on completion, log which class it was. A class that hits the ceiling repeatedly is either misrouted or genuinely underfunded, and those need opposite fixes.

Do This Next Week

Pull the last 30 days of agent runs and sort by cost, descending. Look at the top 20. For each one, answer whether it shipped. That is a single afternoon and it will tell you whether your expensive tail is your best work or your worst, which determines whether the correct control is a cheaper route or a higher allowance.

Then take the highest-volume machine-issued path you run, most likely PR review or a scheduled job, and route it by task class rather than by a global model default. Scoping is free there because the task statement is already written by a system. Watch the end-to-end completion rate for two weeks before you touch anything a human types into a chat box.

One reason to keep the routing layer rather than settle on a model: 33 new models were released in 2026 alone. Any static choice is a decision with a short shelf life. The layer that reclassifies and reroutes is the asset. The model behind it is the replaceable part.

The agent budget that survives the next year looks like a per-run ledger with a policy attached, and the policy needs permission to spend more as well as less. Who holds that permission is its own question, and it is worth settling before the ledger exists.

This analysis synthesizes More models, more choice: comparing 11 different AI models (Netlify, August 2026) and Smart Routing in Unity AI Gateway: match frontier quality with 30%+ lower cost per task (Databricks, August 2026). Both are vendor publications with a commercial interest in the conclusion, and their benchmark figures come from experiments that cannot be independently verified.

Victorino Group helps engineering organizations instrument per-run agent cost and build routing policies that authorize spend in both directions. Let’s talk.