The Classifier Threshold Nobody Signed Off On

On the IBM-Telco churn dataset, the default 0.5 decision threshold prices about $86.11 of avoidable spend per customer. Across 100,000 subscribers, that is $8.6M a year, decided by a single comparison operator that no one in finance ever approved. The number comes from Fabio Oliveira’s analysis in Towards Data Science, and it exposes a governance artifact hiding in plain sight: predict(X) >= 0.5.

That line is where a probability becomes an action. Below the number, the customer is left alone. At or above it, the business spends money to retain them: a discount, a call, a concession. The threshold decides who gets treated and who does not, and every one of those treatments has a cost. A data scientist typed 0.5 because it is the library default, and the model went to production carrying a pricing rule that finance would never have signed if it had been written on a term sheet.

The Threshold Is a Business Decision Wearing a Math Costume

0.5 looks like a neutral midpoint. It is anything but neutral. It only makes sense when the cost of a false positive equals the cost of a false negative, and those costs are almost never equal.

In the churn case, a false negative means the model missed a customer who was about to leave. You lose their lifetime value. A false positive means the model flagged a loyal customer who was never going to leave, and you spent retention budget on someone who did not need it. In Oliveira’s accounting on this dataset, missing a churner costs roughly 13.2 times more than over-treating a loyal one. The false-negative to false-positive ratio is 13.2:1.

When one error is 13 times more expensive than the other, 0.5 embeds a systematic bias toward the cheaper mistake in the wrong direction. The model waits until it is more than 50% sure a customer will churn before acting, while the economics say you should act on the faintest signal, because the downside of inaction dwarfs the cost of a wasted discount.

Anchor the Costs and the Boundary Moves

Once you price the two errors in real dollars, the cost-optimal threshold stops being a matter of taste and becomes arithmetic. You sweep the threshold from 0 to 1, compute total expected cost at each point using the actual false-negative and false-positive dollar values, and pick the minimum.

On this dataset, the cost-minimizing threshold lands at 0.03, not 0.5. Acting when the model is only 3% confident sounds reckless until you remember the 13-to-1 ratio: at that asymmetry, treating almost everyone who shows a whisper of churn risk is cheaper than missing the few who leave. Moving the threshold from 0.5 to 0.03 recovered $121,160 on a test set of 1,407 rows. Scale that to the full subscriber base and the $8.6M figure stops looking like a rounding error.

Two costs drive the whole calculation, and both belong to finance, not to the model:

Customer acquisition cost, which sets what a lost customer is worth to replace.
Survival-based lifetime value, which estimates how long a retained customer keeps paying, using survival analysis rather than a flat average.

Neither number lives in the notebook. Both live in the business. The threshold is where they get spent, which is exactly why the person who owns the budget should own the threshold.

Why Nobody Signed Off

Here is the uncomfortable part. Of 36 indexed public analyses of this same IBM-Telco dataset, 80 to 90% report F1 score or AUC. Under 15% ever draw a profit curve. Zero compute a survival-analysis lifetime value. The field optimizes for a leaderboard metric and stops before the dollar question.

F1 and AUC are threshold-agnostic or threshold-averaged. They tell you the model ranks customers well. They say nothing about where to cut the ranking, and the cut is the entire business decision. A model with a beautiful AUC and a default threshold can still bleed millions, because AUC never asked what a false negative costs.

So the threshold ends up unsigned by default. The data scientist sees a modeling parameter and picks the library value. Finance sees a model they were told is “94% accurate” and assumes the money question was handled. Product sees a dashboard. No one sees the pricing decision, because it is disguised as >= 0.5 inside a function call, and functions do not show up in budget reviews. This is the same failure we described in the deterministic shell around probabilistic agents: the consequential control lives in code, not in a place any owner is watching.

Make the Threshold a First-Class Governance Artifact

The fix is not a better model. The model is fine. The fix is to lift the threshold out of the code and give it a name, a dollar rationale, and an owner.

Treat every deployed scoring model the way you would treat a pricing change. A churn threshold, a fraud threshold, a credit-risk cutoff, a content-moderation boundary, a lead-scoring line: each one converts a probability into a dollar-denominated action, and each one is currently set by whoever wrote the predict call. That is an ungoverned control surface, the same category of risk as an unreviewed config value that quietly moves money.

A governed threshold has four properties. It is written down as a number with a date. It carries the false-negative and false-positive costs that justify it. It names a business owner, typically in finance, who approved the trade-off. And it is reviewed on a schedule, because acquisition cost and lifetime value drift, and a threshold that was optimal in Q1 prices the wrong amount by Q4.

Do This Now

Pick one scoring model you have in production. Find the line where its probability becomes an action. It will be a comparison against a constant, almost certainly 0.5.

Then ask three questions and write down the answers:

What does a false negative cost us in dollars, and what does a false positive cost? If no one can answer, that is the finding: you are pricing a decision with no price.
Sweep the threshold across its range against those two costs. Where is total expected cost lowest? If it is anywhere other than your current value, you have quantified the money the default is costing.
Who signs off on the number? Put a name on it, ideally the person who owns the budget it spends. Add the threshold and its cost rationale to whatever your team reviews when it reviews pricing.

The exact numbers here belong to one dataset and one author’s cost assumptions. The structure generalizes to every scoring model you run. A probability threshold is a price. Right now it is set by a default and signed by no one. Give it an owner before your next incident review discovers it for you.

This analysis synthesizes Your Churn Threshold Is a Pricing Decision (Towards Data Science, June 2026). The specific figures come from the IBM-Telco churn dataset and the author’s cost assumptions, not universal benchmarks.

Victorino Group helps teams turn ungoverned model thresholds into priced decisions with a named owner. Let’s talk.