OpenAI Recommended the Benchmark. Then It Audited It and Found a Third of the Questions Wrong.

Eight months. That is how long it took the frontier pass rate on SWE-Bench Pro to climb from 23.3% to 80.3%. A jump that steep is either a real capability leap or a broken measuring instrument. OpenAI audited the benchmark to find out, published the result on 8 July 2026, and the answer was the instrument. Roughly a third of the benchmark’s public tasks are defective.

Two independent counts on the 731-task public split landed on the same order of magnitude. An automated pipeline flagged 200 tasks, 27.4% of the split. A separate human annotation campaign flagged 249 tasks, 34.1%. An initial automated filter had surfaced 286 candidates for deeper review before the number settled.

We have written about benchmark trust from three angles already. Contamination is the model having seen the answers during training. Infrastructure inflation is the scaffold pumping up the score around an honest question. The scoreboard broken at both ends is mis-measurement at input and output. This audit exposes a fourth failure that sits underneath all three: the questions themselves are invalid. A test that fails a functionally correct answer measures nothing, however clean the training data and however honest the harness.

A vendor auditing its own recommendation

OpenAI had recommended SWE-Bench Pro. After running the audit it withdrew that recommendation in writing. A vendor retracting a benchmark it endorsed is rarer than it sounds, and worth reading closely.

The stated reason is governance. OpenAI treats the retraction as process hygiene. These evaluation results feed OpenAI’s Preparedness Framework decisions, the internal process that gates model releases against capability and safety thresholds. When the eval is defective, in OpenAI’s own words, it “misrepresents safety cases.” A benchmark that lets an incomplete fix pass does not just flatter a model. It tells the people deciding whether a system is safe to ship that the system cleared a bar it never touched.

The standard OpenAI sets for a usable eval is compact enough to steal: an eval should be “hard to game, easy to trust, and genuinely reflective of model capability or alignment.” Three properties. Most internal eval suites at most companies satisfy the first by accident and the other two never.

One correction before the taxonomy, because two newsletters got it wrong. TLDR DEV credited this audit to Anthropic; TLDR AI credited OpenAI. Both linked the identical openai.com URL, differing only by a UTM tag. The publisher and author is OpenAI. The likely source of the mix-up: OpenAI’s audit tooling ran on Codex-based investigator agents, and a reader skimming for a name found the wrong one. If you saw this credited to Anthropic, that was the newsletter, not the source.

The four ways a question breaks

The audit sorts every defect into four categories. This is the part you keep. It is a checklist you can run against your own tasks.

Overly strict tests. The hidden tests enforce implementation details the prompt never specified. A submission that solves the stated problem correctly still fails, because it chose a variable name, a return shape, or an internal structure the test author silently demanded. The model was right. The test was wrong.

Underspecified prompts. The mirror image. The prompt omits requirements that the hidden tests then enforce. The model cannot satisfy a constraint it was never told about. No amount of capability closes that distance, because the information needed is absent from the question.

Low-coverage tests. The tests under-check the feature, so an incomplete fix passes. This is the dangerous one, because it inflates scores upward and silently. It is also where machine and human reviewers diverged most: the agent pipeline flagged 4.1% of tasks as low-coverage, human engineers flagged 9.4%. Humans caught more than twice what the automation did. The automated pipeline systematically undercounts the exact defect that makes a benchmark look better than it is.

Misleading prompts. The prompt points the model toward the wrong behavior, directly contradicting what the tests demand. The task is a trap. Following the instructions guarantees failing the grader.

Two of these four (overly strict tests and misleading prompts) punish correct work. Two of them (underspecified prompts and low-coverage tests) let the wrong work through. A benchmark carrying all four is not measuring capability. It is generating noise with a decimal point.

The protocol that found them

The finding is useful. The method is the transferable asset, and it is cheaper to copy than to admire.

Stage 2b of the audit put every task in front of five experienced software engineers. Each was trained on the four-category taxonomy first. Each judged independently from the problem statement, the tests, and the gold patch, forming a verdict before seeing any pipeline output. Disagreements were escalated rather than averaged away.

The results are what make the protocol worth adopting. Agent-versus-human category agreement ran at 74%. In no flagged task was “not broken” the most common human verdict. Every task the pipeline flagged, the human panel also considered defective. The automation and the humans disagreed only on which defect each task carried. On whether a defect was present, they agreed. That shared verdict is the signature of a real problem, the kind a single picky reviewer could not manufacture alone.

Notice the ordering rule, because it is the part teams skip. Reviewers formed a judgment from the raw materials before seeing the machine’s flag. Show the flag first and you get confirmation dressed as review. The humans anchor, then the automation checks them, then the humans adjudicate.

Run this against your own evals this week

Every company shipping AI features now runs internal evals to decide what is good enough to release. Almost none of them audit those evals. The SWE-Bench Pro numbers are a warning about what an unaudited suite hides: a third of your pass-fail signal may be measuring the wrong thing, in both directions at once.

Pull ten tasks from your most-trusted internal eval. For each one, without looking at any automated score, have an engineer read the prompt, the tests, and the reference solution, and answer four questions. Do the tests enforce anything the prompt never asked for? Does the prompt omit anything the tests require? Could an incomplete solution pass? Does the prompt point away from what the tests want? Every yes marks a defect in your ruler, and the ruler is what you fix.

Then do the arithmetic that matters. If two of ten tasks are broken, and your release decision rides on a two-point score movement, your release decision is riding on noise. OpenAI needed five trained reviewers and a taxonomy to trust its own benchmark. Your internal suite, the one gating your production model, has almost certainly never been checked at all.

A model’s score is only as trustworthy as the questions behind it. This week, read your questions.

---

This analysis synthesizes Separating Signal From Noise in Coding Evaluations (OpenAI, July 2026).

Victorino Group helps engineering organizations audit the evals their release decisions depend on, before those decisions ship. Let’s talk.