Stop Scoring Models on Their Benchmark. Mine Your Merge Queue for Yours.

Two teams published coding benchmarks this quarter, and neither one used a public dataset. Ramp built 80 tasks out of its own merged production pull requests. Epoch AI, working with METR, built MirrorCode, where a model must reimplement an entire program and reproduce its output exactly against tests it never sees. Both set an acceptance bar harsher than any leaderboard in circulation. Both published the pipeline that produces the tasks, which is the part you can steal.

We have argued at length that public coding benchmarks are contaminated, invalid by construction, broken at both ends of the measurement, and gamed by the vendors who score on them. That case is made. What follows is the build instruction for the replacement, assembled from what these two teams disclosed.

Your task source already exists, and it is the merge queue

Ramp’s tasks come from pull requests that already shipped across card authorization, bill pay, reimbursements, accounting, procurement, treasury, fraud, and their own agent systems. Eight production domains, none of them toy repositories.

The reason this works is structural. A merged PR already carries every component a benchmark task needs, written by people who had no idea they were authoring an evaluation. The linked issue or description is the prompt. The diff is the reference solution. The tests that ran in CI, plus the human review that let it land, are the grader. The work is extraction rather than authoring, over material that was already specified, already solved, and already accepted by a reviewer who cared about the outcome.

That provenance also fixes the distribution problem. SWE-Bench and its descendants sample from popular open-source Python repositories, which is a strange proxy for a fintech ledger, a claims pipeline, or a Rails monolith with fifteen years of accumulated domain rules. Your merge queue is a sample from exactly the distribution you care about, because it is that distribution.

Discard every task that carries no signal

The filter Ramp describes is the part most internal eval suites never implement, and it costs nothing to adopt.

“When no model solves the task, it could mean a brittle test or broken environment over real difficulty. When every model solves it, the task is discarded for carrying no signal.”

Two rules, pointing in opposite directions. Universal failure is treated as suspicion of your own harness before it is treated as evidence about the model. Universal success is treated as a dead task and removed. What survives is the band where models actually differ, which is the only band where a score can inform a decision.

Run that filter on your internal suite and expect it to shrink. A suite where the frontier models all score 94% tells you the tasks expired, and tells you nothing about which model to deploy.

Four defect codes to run before a task counts

Ramp names four failure modes in its own task construction: test_overconstrained, test_missing_assertion, prompt_solution_leakage, prompt_missing_context.

Read them as a checklist rather than a taxonomy. Two of them punish correct work: an overconstrained test rejects a valid solution for choosing a different internal structure, and a prompt missing context demands behavior the model was never told about. Two of them inflate the score: a missing assertion lets a partial fix pass, and solution leakage hands the answer to the model inside the prompt itself.

Leakage is the one unique to this build path, and it is the one you will hit. When you generate a prompt from a merged PR, the description frequently contains the fix. “Cache the rate lookup in the session” is not a task statement. It is the diff, in prose. Every task you mine needs a pass where a human strips the solution back out and leaves only the observable problem: the bug report, the failing behavior, the requirement as the requester stated it before anyone knew how to solve it.

The overlap with OpenAI’s SWE-Bench Pro audit is worth noticing. Two organizations, working independently on different corpora, converged on essentially the same four ways a coding task breaks. That convergence is a strong signal these are the real failure modes, not an artifact of one team’s methodology.

Set the bar where your engineers actually set it

Ramp scores pass@1 on a single attempt. No retries, no best-of-n. Running out of context window counts as a failure, not as an incomplete run to be discarded.

Their stated justification: “Each result is a single pass@1 attempt, mirroring the bar engineers hold background agents to: a correct, review-ready change on the first try.”

That is the acceptance criterion doing real work. A pass@10 number tells you a model can eventually produce a correct patch if a human keeps rerolling. That is a fine research metric and a useless procurement metric, because nobody staffs a person to reroll an agent ten times per ticket. Counting context exhaustion as failure closes the other escape hatch: a model that cannot finish inside its own window has not solved your task, whatever the partial diff looks like.

Ramp’s interactive leaderboard did not render when we pulled the page, so we have no model scores to report from it. The methodology is the transferable asset anyway.

The harder variant: an oracle the model never sees

MirrorCode takes a different route to the same goal. The model gets a program’s documentation and must reimplement it from scratch, in a specified language, well enough that its output matches the original exactly. The tests are held out. There is nothing to pattern-match against, because the acceptance criterion is behavioral equivalence with a reference implementation the model cannot read.

The leaderboard configuration (15 targets, two languages, three runs each, with 10 billion tokens and seven days allowed per attempt) reports solve@100% of 64% ±10 for Claude Fable 5, 20% ±9 for GPT-5.6 Sol, 16% ±8 for GPT-5.4, and 10% ±6 for GPT-5.5. Epoch’s paper used a different configuration and states those numbers are not comparable, so treat this leaderboard as one measurement, not as a ranking you can port elsewhere.

The budget numbers are where this gets operationally interesting. Claude Opus 4.7 reimplemented gotree, roughly 16,000 lines of Go with more than 40 commands, in 14 hours for $251, against a human estimate of two to seventeen weeks. The best runs passed 2,000 of 2,001 tests. The largest task in the set cost $2,600 for a single run, with the model working 19 days without human intervention.

Epoch’s observation about why nobody sees this: “Many existing software engineering benchmarks limit inference spending to around $1-10, even when the task would take weeks for a human to complete.” A ten-dollar cap on a two-week task is not measuring capability. It is measuring the cap. If your internal eval imposes a token budget you picked for convenience, you are publishing a number about your budget and labeling it a number about the model.

Two caveats Epoch states plainly, and you should carry both. Even failed attempts usually pass 90% or more of tests, so a binary solve rate hides most of the actual signal. And they cannot rule out that memorization contributes to performance, since the reference programs are public.

The data has to be contractually excluded from training

Ramp closes the loop with one sentence: “Ramp SWE-Bench data is never used for training, enforced through agreements with model providers.”

This is the difference between a benchmark and a benchmark that stays valid. The instant your tasks reach a training corpus, the next model generation scores well on them for the wrong reason, and you have rebuilt the problem you left the public leaderboards to escape. A technical control is not enough here, because you are sending the prompts to someone else’s inference endpoint. The control is contractual, and it belongs in the same vendor agreement where you negotiate retention and data residency.

If your provider contract does not currently say your evaluation data is excluded from training, your internal benchmark has a shelf life measured in model releases.

Build the first version this sprint

Pull your last 200 merged PRs. Keep the ones with a linked issue, a test change, and a diff under roughly 400 lines. That will leave 30 to 50 candidates, which is enough to start.

For each candidate, strip the solution out of the prompt and keep only the problem as it was stated before the fix existed. Then run the four defect codes against it and throw away everything that fails. Score three models pass@1, single attempt, with context exhaustion counted as failure. Delete every task all three solve and every task none of them solve, then check the ones nobody solved for a broken environment before you believe the difficulty.

What you have left is small, ugly, and yours. It will predict which model handles your codebase better than any public score ever has, for the plain reason that it is made of your codebase. Then call your provider’s account team and get the training exclusion in writing, because that clause is what keeps the whole thing measuring something a year from now.

---

This analysis synthesizes Ramp SWE-Bench (Ramp Labs, July 2026) and MirrorCode: What’s the largest software project AI can complete on its own? (Epoch AI, co-developed with METR, June 2026).

Victorino Group helps engineering organizations build internal benchmarks from their own production history, so model decisions rest on evidence from the codebase that pays the bills. Let’s talk.