The Oracle Test: When an Agent Can Be Trusted to Find a Bug

Mean time-to-exploit is now negative seven days. Mandiant’s M-Trends 2026 puts it plainly: vulnerabilities are often exploited a week before a patch even exists. In the intrusions Mandiant investigated last year, vulnerability exploitation was the primary initial infection vector.

That pressure is why every security team has been told to point an agent at its codebase. It is also why so many of those deployments produce a queue nobody trusts. Mandiant Consulting’s AI-Assisted Vulnerability Management, bylined by Jules Czarniak with the Google Threat Intelligence Group, contains the one primitive that explains the difference. It is a question about the bug, not about the model.

The Question: What Proves It?

An oracle is whatever tells you a finding is real. Every automated bug-finding system depends on one, and the properties of that oracle decide whether an agent can operate on its own.

The source states the mechanism directly: “if an agent’s payload fails to produce a clear outcome, it can’t reliably distinguish whether the vulnerability is a hallucination or if it simply constructed the payload incorrectly.” Two failures collapse into the same silence. The agent cannot tell them apart, so it either guesses or reports both. This is why agentic scanning of enterprise code produces confident findings at volume with no way to grade them.

So the test to run against any bug class you were planning to hand an agent: if the agent is right, what happens within seconds, observably, without a human interpreting it?

Class One: The Oracle Is Binary

Memory corruption answers that question with a crash. Send the payload, the process dies, the sanitizer prints a stack trace. Signal arrives in milliseconds and requires no judgment. Fuzzing has worked for two decades on exactly this property, and the reason agents amplify fuzzing so effectively is that they inherited a working oracle rather than inventing one.

For this class, autonomy is a straightforward engineering decision. The agent proposes, the harness proves, and the volume of wrong proposals costs almost nothing because each one is settled automatically.

Class Two: The Oracle Is Architectural

Now take an authorization bypass. The agent crafts a request that reads another tenant’s records. Nothing crashes. The response is a normal HTTP 200 with a normal JSON body. Deciding whether that body represents a breach requires knowing which tenant owned the record, which role should have been able to see it, and what the intended policy was. That knowledge lives in a design document, a compliance matrix, or somebody’s head.

Business logic flaws behave the same way. So does indirect SSRF, where the observable outcome is a request the agent cannot watch land. The source is blunt about the distribution: enterprise software “is heavily dominated by vulnerabilities that require architectural oracles.”

Set that against the headlines. The publicized wins in AI vulnerability discovery cluster in memory-unsafe C, parsers, codecs, and kernels, which is precisely where the crash oracle lives. The bug classes that actually breach an enterprise cluster in the other half. Google’s CodeMender sits on the binary side of that line, and the source describes it as not publicly available, so it is a signal about direction rather than a tool you can adopt this quarter.

The practical consequence is uncomfortable but clean. For architectural oracles, the agent’s output is unfalsifiable at machine speed. A human owns the finding from the moment it appears, and the agent’s role shrinks to surfacing candidates for a person who already understands the intended design.

The Gate: Discard Before a Human Sees It

Most triage architectures place the human first and the proof second. The source inverts that, and this is the part worth stealing even if you take nothing else.

The agent is required to produce “a fully reproducible, deterministic test harness (such as a compiled binary or a Python test script) that attempts to prove the exploit.” That harness runs automatically in an isolated, monitored sandbox with hard execution timeouts and iteration limits. If the sandbox cannot prove the exploit, the ticket is discarded. No human ever opens it. If the sandbox proves it, a human engineer reviews a finding that already comes with a working reproduction.

The economics behind this are stated just as directly: “LLMs can generate findings significantly faster than human engineers can triage them.” A queue that grows faster than it drains is not a security program. Automating the proof step, and letting it delete rather than rank, is what keeps human attention scarce and therefore useful.

The same script earns a second job later. “By rerunning the initial PoC against the patched code, the workflow repurposes the exploit script as a validation oracle to prove the vulnerability has been remediated.” One artifact closes the loop it opened.

Three controls hold the pipeline together, and each one fails quietly if skipped. CI/CD access is “restricted to generating PRs, never direct commits to main.” Machine identities are least-privileged and tied to a named human controller, with short-lived JIT tokens “bound exclusively to the specific repository and branch.” Model API versions get pinned to frozen releases, because a silent weight update will change agent behavior against a harness that was calibrated on the old one. Immutable audit logs then record which model version proposed a fix, which deterministic tests validated it, and which engineer approved the merge, which is the evidence SOC 2, PCI-DSS, FedRAMP, CMMC, and the EU AI Act human-oversight expectations all end up asking for.

For findings that clear the gate, the source offers a risk-based scoring formula worth keeping as a compact artifact:

Final Score = (W1 x S_vuln) + (W2 x S_asset) + (W3 x S_threat)
Sample weights: 0.20 / 0.40 / 0.40, scale 0-100

S_vuln is CVSS Base multiplied by 10. S_asset runs from 100 (internet-facing with customer data) down to 25 (internal, no sensitive data). S_threat is 100 when actively exploited by relevant actors, 75 when a PoC exists or the class is easily exploited by autonomous AI agents, and 25 when theoretical and highly complex. The interesting move is the weighting: asset context and threat context together carry 80% of the score, so CVSS stops driving the queue.

Where Agents Fail Silently

Five failure modes are named, and all five are invisible without a proof step in front of them.

Attention degradation: a variable tainted on line 10 and sanitized on line 500 gets scored as tainted. Contextual narrowing: the agent hyper-fixates on a SAST-flagged syntax error and misses architectural flaws in the same file. Novel techniques produce silent false negatives, since “the agent will likely scan right past it in silence.” Stale RAG makes the agent confidently describe “a secure path that no longer exists in production.” And runaway iteration loops burn budget without converging, which is why the timeouts are mandatory rather than advisory.

Two supply-chain notes belong next to those. The guidance that “security teams should treat the codebase itself as an untrusted input” follows directly from prompt injection living in comments and test fixtures. And “MCP plugins introduce the risk of supply chain poisoning, where a previously benign integration is silently updated with malicious dependencies.” We have covered the containment surfaces this implies in the agent containment stack and in five levels of bash containment; the oracle test sits upstream of all of them, deciding whether the agent should be pointed at that code at all.

Do This Now

Take your last twenty security findings, from any source, and sort them into two columns: those where a script could have proven the finding in under a minute without human interpretation, and those where proving it required knowing the intended design.

Point agents only at the first column. The source narrows the targets to four candidates: memory-unsafe codebases, systems exposed to outside content, shared internal libraries, and foundational security boundaries. Its own framing is a warning against breadth: “Rather than indiscriminately pointing agents at all available codebases,” restrict where they run.

For the second column, the honest answer is that agents help a human read faster and cannot own the verdict. Staffing the review is the control. Buying a scanner is not.

One caveat on the source itself. It is Mandiant advisory content with Google SKUs attached, including Model Armor, Assured OSS, the Mandiant Threat Modeling Security Service, and Wiz. The oracle distinction holds independently of any of them. The product recommendations carry a commercial interest and should be evaluated as such. Also worth noting: when AI finds real vulnerabilities, disclosure and ownership become governance questions well before they become engineering ones, and the architecture problem does not resolve itself with a better model.

This analysis synthesizes AI-Assisted Vulnerability Management (Mandiant Consulting / Google Cloud, Jules Czarniak with the Google Threat Intelligence Group, July 2026), with the mean time-to-exploit figure originating in Mandiant’s M-Trends 2026 report.

Victorino Group helps engineering organizations decide which bug classes their agents can own and builds the deterministic proof gates that make the rest safe to triage. Let’s talk.