Three of Four AI Security Patches Are Broken. 6,080 Patches Say So.

Off-by-1 Labs, 1Password’s security research team, generated 6,080 patches for six recently-disclosed CVEs using two frontier reasoning models. Then they measured what those patches actually did. Only 26.0% completely fixed the vulnerability without changing application behavior. Another 20.1% fixed it and broke something else. The remaining 53.9% failed to fix the flaw, introduced a new vulnerability, or managed both at once.

The researchers went in expecting a success rate above 67%. They published the number they got instead.

What Was Actually Measured

The setup matters, because the temptation is to dismiss a bad result as a bad prompt. Off-by-1 ran 540 patch attempts per vulnerability per model, across three environment configurations and nine per-CVE prompt templates. ChatGPT-5.5 ran at medium reasoning effort with Trusted Access for Cyber enabled. Anthropic’s Opus 4.8 ran at high effort under the Cyber Verification Program. Both models had the vulnerability report, the code, and the ability to iterate.

The six targets are not toy bugs. CVE-2026-31431 is a Linux privilege escalation. CVE-2026-34197 is remote code execution in ActiveMQ. CVE-2026-8512 is a use-after-free in Chrome’s File System Access API on macOS. CVE-2026-45185 is unauthenticated RCE in EXIM. CVE-2026-22738 is SpEL injection in SpringAI. GHSA-wpqr-6v78-jr5g is RCE in the Gemini CLI. Six real advisories, spanning kernel, message broker, browser, mail server, and two AI-adjacent codebases.

Every attempt was scored against a five-scenario taxonomy: complete fix with no behavior change (S1), complete fix that altered behavior (S2), no fix (S3), fix plus a new vulnerability (S4), and no fix plus a new vulnerability (S5). The last three are what the team named FLAWED, Fix-Like Artifacts With Embedded Defects. That category is the majority of the output.

The cost of producing that output is worth writing down, because it is small enough to be dangerous. $2.11 per attempted patch and validation cycle for ChatGPT-5.5. $2.81 for Opus 4.8. At those prices an organization can generate remediation faster than it can evaluate it, which is the exact condition under which unverified work accumulates.

The Asymmetry Nobody Priced In

Keith Hoodlet, who authored the research, states the core finding directly: “LLMs that excel at discovering a wide range of vulnerabilities today are only currently effective at patching a narrow subset of them.”

Discovery and remediation are two different capabilities that happen to live in the same model. Finding a vulnerability is a search problem with a verifiable endpoint: the exploit either fires or it does not. Fixing one is a design problem constrained by everything the code was already doing correctly. A model can be excellent at the first and mediocre at the second, and nothing about its performance on the first tells you anything about the second.

The industry has been quietly assuming otherwise. Anthropic’s May 2026 Project Glasswing update framed the shift honestly: “Progress on software security used to be limited by how quickly we could find new vulnerabilities. Now it’s limited by how quickly we can verify, disclose, and patch.” Every AppSec roadmap that treats AI-assisted remediation as the natural follow-on to AI-assisted discovery inherited an assumption that this data does not support.

The SpringAI case shows the failure mode with unusual clarity. Patches escaped specific characters in user input while leaving the injection root cause untouched. The vulnerability scanner goes quiet. The exploit path is still there, waiting for a slightly different character. The researchers note this “would lead to the old vulnerability resurfacing in the software.” A closed ticket, an unclosed hole, and no signal anywhere in between.

Reading the Diff Is Not a Control

More than 33% of the patches that succeeded, the S1 and S2 outputs, still contained subtleties the researchers judged fragile from a security standpoint. So the good pile has its own contamination rate. Passing the automated check is not the same as being sound.

Off-by-1 gave the human half of this problem a name: cognitive surrender. A reviewer who is not paying careful attention approves LLM code because it looks like a fix. It has the shape of a fix. It touches the file the advisory pointed at. It adds a sanitization call. It reads as competent, because the model that wrote it is genuinely competent at producing text that reads as competent.

This is why a review process built on reading diffs is not a control for AI-generated security work. Reading is exactly the channel the failure mode is optimized to pass. We have argued before that agents will game verification once it operates at scale; this is the same dynamic without any adversarial intent, arriving through sheer plausibility. And it is why we keep insisting that AI governance is a cybersecurity discipline rather than a documentation exercise. A policy document does not run the exploit. A test harness does.

The control point is execution-grounded verification. The patched build has to be attacked with the original proof of concept and confirmed dead. The application’s behavior has to be exercised against a regression suite that catches the S2 failures, the ones where the vulnerability is gone and so is a feature. Neither of those is a reading activity. Both are runnable, which means both are automatable, which means both scale with the generation volume that $2.11 per patch makes possible.

What Makes This Different From an Opinion

Off-by-1 Labs released the FLAWED tooling and the datasets. Any team can point the same measurement at its own codebase, its own CVEs, and its own model configuration.

That changes the shape of the internal conversation. The question stops being whether AI patching is good enough, argued from vendor claims and individual war stories, and becomes what your own S1 rate is on your own code. Different languages, frameworks, and vulnerability classes will produce different numbers. Some teams will beat 26.0%. Some will not come close. Either way the answer is measurable in an afternoon, and an organization that has measured it is negotiating with a number while its competitors are negotiating with adjectives.

This is the same discipline we described for code quality baselines, applied to a domain where the cost of being wrong is an unclosed vulnerability rather than a slower sprint.

Do This Now

Take the last ten security fixes your organization merged with AI assistance. For each one, answer a single question with evidence: did anyone run the original proof of concept against the patched build?

If the answer is no for most of them, you do not have a remediation process. You have a generation process with a reading step attached, and the published data says that step approves the majority of broken work. Pull the FLAWED tooling, run it against one representative service, and put the resulting S1 rate in front of whoever signs off on your security posture. Then make the exploit rerun a merge requirement, not a best practice.

The teams that will handle the next two years well are the ones whose patch pipeline can prove a vulnerability is dead, in an environment, on demand. Everyone else is shipping fixes they have only read.

---

This analysis synthesizes Off-by-1 Labs Research: AI-generated vulnerability patches require human review (1Password / Off-by-1 Labs, Keith Hoodlet, August 2026) and the FLAWED tooling and datasets (Off-by-1 Labs, August 2026).

Victorino Group helps engineering organizations build execution-grounded verification into their AI-assisted security remediation pipelines. Let’s talk.