Your Agent's Failure Is a Report on Your Codebase

Almost 16,000 merges have been withheld inside Cloudflare by a system called Codex, which reads pull requests against the company’s engineering RFCs. Since inception earlier this year it has flagged close to 230,000 violations. Only a subset of those violations carry the power to stop a merge, and which subset is a decision a human made deliberately, RFC by RFC.

That distinction is the whole product. In the same week, Matthew Phillips published how the Astro maintainers drove their open issue count from over 200 to about 30, expecting zero, the first time the repository has seen zero open issues in its 5+ year history. Two different problems, two different companies, and the same underlying move: both teams treat what an agent produces as a measurement of the organization rather than a verdict on the model.

Approved and enforced are two different states

Cloudflare’s RFC corpus is what most engineering organizations already have in some form. Design documents, standards pages, an architecture wiki. What they added is a state machine on top of it. A standard can be approved, meaning the organization agrees with it. A standard can be enforced, meaning a violation of its MUST statements withholds approval on a pull request.

Writing the standard is not the governance act. Promoting it from approved to enforced is. That promotion is the moment a human accepts the operational cost of the rule: the merges it will block, the arguments it will start, the exceptions it will require. Almost 16,000 withheld approvals is the price tag on that decision, made visible.

Most teams that adopt AI code review skip this. They point a model at a style guide and let it comment on everything, which produces a review surface nobody can calibrate. The advisory findings and the blocking findings blur into one stream of suggestions, and engineers learn to scroll past all of it. Splitting the two states gives the organization a dial it can actually turn, one RFC at a time, with a measurable consequence attached to each turn.

We argued in encoding team standards as governance that instruction files are developer tooling and deserve the same review discipline as code. The promotion step is what that discipline looks like when it reaches production. It is also the answer to the objection raised in governance belongs in the backlog: enforcement has a cost, and a state machine is how you schedule paying it.

The corpus does not go in the context window

Cloudflare has 60+ RFCs and counting. They rejected the obvious implementation, which is to hand the whole corpus to the model and ask it to judge a diff. Their stated reason is that the corpus volume would put a lot of stress on the context window.

The design they shipped instead: each RFC is parsed into individual normative statements, extracted to JSON, and each statement carries a stable slug. The reviewer evaluates a pull request against a selected set of statements rather than against a pile of prose. Because the slug is stable, a violation recorded in March remains identifiable in August even after the RFC around it has been rewritten.

That stability is what makes the 230,000 number a measurement instead of a log. You can ask which statement fires most, which one produces the most overrides, which one was promoted to enforced and then quietly generated a backlog of workarounds. A corpus stuffed into a prompt gives you none of that. It gives you an opinion per diff and no way to compare opinions across time.

Cloudflare runs the same architecture on two other surfaces. A spec reviewer has evaluated close to 600 technical designs across over 3,200 review invocations since May 2026, with findings landing at major (65%), minor (29%) and critical (6%). An incident report reviewer has assessed more than 200 reports, and the interesting figure there is not the volume: 93% covered incidents that were low-impact, internal-only, or declared preemptively. That is a statement about which incidents an organization writes up, and it came out of a review pipeline that was not built to answer it.

Astro’s failure taxonomy points at the repository

The Astro software factory runs four isolated subagents over an incoming GitHub issue: Reproduce, Diagnose, Verify, Fix. Each passes a report.md forward to the next. The isolation is deliberate, and Phillips states why: to prevent the frequent LLM bias toward forcing a solution when a bug might not actually exist. An agent that has spent 40 turns diagnosing will find a fix whether or not one is warranted. Cutting the context between stages removes the sunk cost.

The system holds almost no state. Two labels, triage needed and fix verified. The issue thread is the memory. The original reporter confirms the fix before a pull request opens, which means the acceptance signal comes from the person who has the failing case, not from the agent that wrote the patch.

The part worth stealing is what happens when the pipeline fails. Astro classifies every agent failure into one of three codebase defects: opaque abstractions, missing documentation, insufficient testing. Phillips puts the inference plainly: “If an agent cannot interpret the boundaries between components, human developers likely struggle with the code structure as well.”

The fix goes into the repository. Not into the prompt.

The inversion

This reverses the default reflex. When an agent produces something wrong, the instinct is to tune the instructions, add an example, switch models, raise the reasoning budget. All of that treats the agent as the unit under test.

Read the failure the other way and it becomes free diagnostics on a system you already own. An agent that cannot tell where one module ends and another begins is reporting an abstraction with no observable boundary. An agent that guesses at a function’s contract is reporting undocumented behavior that your new hires are also guessing at, more slowly and less visibly. An agent whose fix breaks something unrelated is reporting a test suite that does not defend the thing it broke.

None of those readings requires trusting the model’s judgment. They only require noticing where it stalls. A junior engineer’s confusion carries the same signal, but you get it once per hire, filtered through their reluctance to look slow. An agent stalls on every run, in a log, at whatever volume you point at it.

This is the complement to the containment argument in the cage pattern. Containment limits what an agent can damage while it works. Reading the failure log tells you what the work revealed about the codebase underneath.

Do this now

Take the last twenty times an agent produced something you rejected in your repository. Not the model’s refusals or hallucinated APIs, the cases where it wrote plausible code that was wrong for your system. Classify each into Astro’s three buckets: the boundary was unclear, the behavior was undocumented, the test did not exist. Count them.

Whichever bucket wins is your next refactoring ticket, and it is backed by evidence that a human would have taken a quarter to produce. If a fourth bucket appears that none of those three describe, name it and keep counting. That is your organization’s own defect taxonomy forming.

Then pick the single standard your team argues about most and ask which state it is in. If everyone agrees with it and nothing enforces it, it is approved. Decide out loud whether it should be enforced, and accept that the answer might be no. A rule nobody is willing to let block a merge is a rule the organization has already declined to have.

---

This analysis synthesizes How Cloudflare enforces engineering standards using AI (Cloudflare, August 2026), How we built a software factory to drive Astro’s GitHub issue count to zero (Cloudflare / Astro, August 2026).

Victorino Group helps engineering organizations turn agent failure logs into a prioritized codebase remediation plan. Let’s talk.