Jev Scored 93%. A Constant Scored 79%. Judge the Gate by How It Fails

The Archestra team took 100 real tool calls from production Claude Code sessions, gave each one four labels, 400 decisions in all, and asked Jev, the decision model from TypeSafe, to classify them. On the 337 where three LLM judges agreed, Jev scored 93%. A constant that always answers “benign” scored 79%. Archestra spells out what that floor means for anyone reading a leaderboard: “A model scoring 75% is worse than that baseline.”

A disclosure before the argument. Victorino uses Jev as a triage gate in its own content-discovery pipeline: it helps decide which articles earn a human deep read. We measured it against our own human triage and got 77% agreement, a figure that appears later in this piece. That use shapes how we read the Archestra results, so weigh our reading with it in mind.

Our position is narrow. When a decision model sits on a control path, where its answer lets an action through, blocks it or routes it somewhere, two numbers matter before any headline accuracy: what a constant scores on the same data, and what the model’s errors look like.

What the 93% Is Measured Against

The Archestra setup deserves a careful read before the numbers do. The ground truth is agreement among three judge families (Claude Opus, GLM 5.3 Flash and Gemini 3.8 Flash). Results are reported on the 337 decisions where all three agreed, out of 400. These are LLM-judge labels rather than human ones. The authors call their results early and say their original benchmark was bad, a useful admission.

On that set, the ranking looks simple:

Majority baseline (a constant): 79%
Jev, 0-shot: 93%
Jev, 9-shot: 95%
Sonnet 5, 0-shot: 98%

Read top to bottom, the table says to pick Sonnet 5. It has the highest accuracy of the four rows. The 79% line changes how much each point is worth. Jev closes about two thirds of the distance between the constant and a perfect score. Sonnet 5 closes more. Both are real improvements over doing nothing, and neither number yet tells you whether the model is safe to put in front of an execution boundary.

We made the general case for this in Govern AI Code Quality on Baselines, Not Anecdotes. A decision model on a control path is the sharpest version of it, because in Archestra’s data the majority answer is the harmless one.

Split the Leaks From the Stalls

Accuracy averages two kinds of error that cost very different amounts. A leak is a dangerous call labelled safe: the action runs. A stall is a safe call labelled dangerous: the action waits for a human. On a control path, the first one is the incident.

Archestra measured refusal recall on the requires_trusted label, the calls that needed trusted handling. There were 9 of them. Jev caught 7. Sonnet 5 caught 4. Of Sonnet 5, the authors write: “It made eight errors on the strict set, and all eight were leaks.”

Keep the denominator in view. Nine calls is a small sample, and 7 against 4 is a difference of three calls. Nobody should rank models on that alone. What it does show is that the more accurate model and the safer model can be different models, and the headline table hid that completely. The minority class is where the danger lives, so it is the class to count on its own, with its own recall.

Waldek Mastykarz offers a test that makes this concrete. For every grader, finish two sentences: “If this grader passes, I now know that ___” and “If this grader fails, I now know that ___”. Then compare them with the claim you intend to make. Try it on “98% accuracy on 337 agreed decisions.” The pass sentence says nothing about the 9 calls that mattered. We wrote about the same ordering question for security review in Precision Before Recall, and about testing the instrument itself in Calibrate the Instrument First.

Test the Confidence Before You Set a Threshold

Archestra also reports a result that makes a gate tempting: “among predictions with confidence of at least 0.7, it made no errors. That does not prove perfect calibration, but it makes confidence-based routing practical.” Their own sentence carries the hedge, and the stability runs explain why it is needed.

Across 5 runs, between 394 and 398 of the 400 labels came back identical on repeat. The probabilities moved more than the labels: drift reached 0.17 on a plain repeat and 0.37 when the answer options were reordered. Option order changed the decision in about 4 of every 100 cases.

Archer Hume found the same sensitivity. Reversing the order of options moved a technical-support probability from roughly 0.84 to 0.89 up to 0.93 to 0.96. Hume’s conclusion is direct: “A threshold near 0.9 could change the action even though the labels and evidence are identical.” The post also points out that the confidence field in the API is computed as c = (p_max − 1/K)/(1 − 1/K). It is a formula applied to the top probability. Nothing in it was learned as an estimate of being right.

The calibration numbers in the same post add a second reason for caution. On a 1,200-item MMLU sample, ten-bin expected calibration error was 0.0313, which looks excellent. Inside the 0.5 to 0.6 bin, predicted accuracy was 52.8% and observed accuracy was 39.3%, on 28 items. A good average can hold a bad band, and a threshold lives in a band. Hume’s recommendation fits any team about to wire a cutoff: “Permutation tests belong in the evaluation of any implementation of this design.”

Let the Model Defer

Tomasz Tunguz ran a different kind of test. On 98 hand-verified production email threads, the production generative classifier scored 47% (46 of 98), Jev scored 80% (78 of 98) and SemIf scored 82% (80 of 98). He reports no constant baseline and no class balance, so the 80% cannot be placed against a floor the way Archestra’s 93% can. The result stands on 98 threads.

The more instructive part is the live run. A local decider acted on 8 of 31 emails with zero errors and deferred the rest. That is the architecture we would copy for a control path: the model handles the calls it is sure of and hands the others to a slower path, whether that means a human, a larger model or a stricter rule. A deferring gate turns uncertainty into a stall, which is the cheap error.

Tunguz also cites a 99% cost reduction. It is list-price arithmetic ($0.042 per million input tokens, output free), “which TypeSafe says may be subsidized.” Treat it as a price that can change. If the design only works at that price, write down what happens when it moves, and how you would swap the model out.

A TypeSafe-side design memo titled “Why yet another agent” (the page carries no author and no date) proposes decision-model calls for command permissions and approvals, and for security-aware routing. It offers no accuracy evidence for that placement. Archestra’s advice applies directly: “Never rubber-stamp a model on your execution boundary based on macro numbers alone.” Simon Willison, writing about Jev, lands in the same place: “evals and structured experiments are even more important than they are for regular LLM projects.” He adds a warning: “I really hope nobody uses Jev to rank job applicants.”

What Our Own Gate Showed Us

Our measurement is small and internal, and the source file lives in our repository rather than on a public page. We compared Jev with human triage on 61 articles. On the deep-read gate, the two agreed on 77% (47 of 61). On relevance, Jev landed within one point of the human score on 84% (51 of 61). Jev would have skipped 13 articles the human chose to read.

Those 13 are our leaks: a skipped article gets no deep read. We gate on Jev’s relevance answer and keep its composite score out of the decision. One pipeline and 61 articles prove little about anyone else’s workload. The method still transfers: measure the gate against human decisions, and count the costly errors on their own.

Do This Now

Before a decision model touches a control path, run this list against your own traffic:

Compute the constant. Report what a constant answer (the majority label, every time) scores on your data, next to every accuracy number.
Split leaks from stalls. Count dangerous-labelled-safe separately from safe-labelled-dangerous. Only the first one is an incident.
Measure recall on the dangerous minority. Give it its own denominator, and say how small that denominator is.
Repeat and permute before you pick a threshold. Rerun the same inputs, reorder the options, and record how far the probabilities move near your cutoff.
Make the model defer. Let it act when confident and route the rest to a slower path. Do not make it the sole gate.
Price the exit. Write down what the design costs if the list price changes, and how you would replace the model.

Then finish Mastykarz’s two sentences for the gate you are about to ship. If the pass sentence does not mention the dangerous calls, the evaluation is not done.

This analysis synthesizes We Tested Jev on 100 Real Agent Calls. How Easy Is It To Beat a Constant? (Archestra, September 2026), Jev’s Architecture Unmasked (Archer Hume, September 2026), AI Comes for the If Statement (Theory Ventures, September 2026), Jev introduces a new shape of LLM: System One, aka Decision Models (Simon Willison, September 2026), Is your eval lying to you? (Waldek Mastykarz, September 2026) and Why yet another agent (unattributed design memo, September 2026).

Victorino Group helps engineering teams evaluate and govern the models that sit on their agents’ control paths. Let’s talk.