design.md Cut Failures 57%. Every Page Still Failed to Ship.

Vercel ran three desktop scenarios through Codex on GPT-5.5, each once with its design.md policy file loaded and once without. Six pages. With the file, deterministic checks counted 39 known failures. Without it, 91. Vercel states that as 57% fewer failures. Then the same post adds the sentence that matters more than the percentage: every one of the six pages, with or without the file, still had at least one failure serious enough to block shipping.

Those two facts belong together. A policy file for an agent raises the floor. It does not lift the ceiling to “ship without a human.” Anyone building agent governance outside code, in finance, legal, procurement, brand, should read the Vercel post for that pair of numbers, and for the loop that produced them.

The post is How our agents build on-brand pages with design.md, by John Phamous with Kevin Corbett contributing, published August 31, 2026. One caveat before the analysis: I am working from a condensed rendering of the page, so the quotes below are the fragments as returned, not full paragraphs. The 57% is Vercel’s own arithmetic from 39 and 91.

What Vercel actually shipped

Two public artifacts. The policy file lives at vercel.com/design.md. The stylesheet lives at vercel.com/geist/vercel-brand.css. The split is clean: the file carries brand judgment as prose an agent reads, and the stylesheet carries bounded mechanics as CSS an agent applies.

Around those two artifacts sits an eval harness. Vercel built seven scenarios from real internal use: a usage and performance report, a renewal proposal, a benchmark report, an interactive planning page, a build-versus-buy brief, a security governance brief, and a presentation deck. None of those is a marketing landing page. They are the documents a sales engineer or a solutions team produces on a Tuesday afternoon, the class of output that, in my experience, rarely passes through design review.

The post reports “more than 200 runs,” a count that includes full rounds, targeted checks and dry runs. The matched comparison that produced 39 against 91 is narrower: three desktop scenarios, one model.

We have covered design.md as an agent-instruction file before, and what happens when agents write the constraint layer themselves. This post is the first place I have seen a company publish the before-and-after counts for such a file. That is the reason to write about it.

The triage rule is the reusable part

Every correction in Vercel’s loop gets sorted into one of three destinations.

A change in judgment goes into design.md as prose. A reusable mechanic goes into the stylesheet. A mechanical failure gets a deterministic code check. The post gives one example that landed in two bins at once: commercial-terms tables were being squeezed to prose width, so the fix was a design.md rule plus a code check. The rule teaches the agent why a commercial-terms table needs room. The check catches the agent when it forgets.

Read the rule as a governance artifact and the design context falls away. Any team that gives an agent a policy document faces the same three questions when the agent gets something wrong. Was this a judgment the agent lacked? Write it down where the agent reads. Was this a capability the agent had to rebuild each time? Package it so the agent applies it instead of improvising. Was this a failure a machine can detect? Then stop asking a human to detect it, and stop asking the agent to promise it will not happen again.

The third question is where I see agent governance efforts stall. Teams write longer and longer instruction files, encoding rules that a regex would enforce more reliably than a model ever will. The Vercel rule pushes the other way: prose for judgment only, code for everything code can hold.

The instruction that keeps the loop honest

One line from the post deserves quoting on its own: “Update the guidance instead of hand-tuning the generated page. Your next comparison tells you whether first attempts actually improved.”

This is the discipline that separates an eval loop from a review queue. In a review queue, a human fixes the artifact and moves on. The agent produces the same defect next week. In Vercel’s loop, the instruction is not to fix the page. The move it prescribes is to change the guidance, and by the triage rule the stylesheet or the checks, then rerun the comparison to see whether the first attempt got better.

The consequence is that every human correction becomes a measurable change to the system, and the system’s improvement is tested rather than assumed. A team that lets reviewers patch outputs directly never learns whether its policy file works, because the shipped result reflects the reviewer, not the policy.

What the ceiling means

Now back to the sentence that matters. Six out of six pages still blocked, with or without design.md.

Vercel says so itself, and adds that six pages is too few for broad quality claims. Take both statements at face value. The sample is small. The direction is clear enough to act on: the policy file removed a large share of the mechanical failures the checks could count, and left intact the class of failure that stops a page from shipping.

That is what a policy file is for. It turns the checkable portion of quality into something the agent gets right on the first attempt more often. The residue, the failures serious enough to block, is the portion that still needs a person. The Vercel result puts a number on that split, and the number says the person is still in the loop after the file is written.

Our earlier piece on agents as drift detectors argued that agents surface where a design system is underspecified. The blocking failures left on those six pages are that residue. Each one is either a rule nobody wrote yet, a mechanic nobody packaged yet, or a check nobody coded yet. The loop exists to move them, one at a time, out of the human’s queue.

The production half

The eval harness is one half of Vercel’s setup. The other half is an internal Slack agent, @design-agent, that produces pages for real requests. Vercel aggregates the repeated complaints weekly and tracks complaint frequency before and after each change to the guidance. Final changes to the guidance are reviewed by a human.

The pairing is the point. The harness answers “did the change help on the fixed scenarios.” The complaint log answers “did the change help on what people actually asked for.” A policy file that improves the benchmark and leaves the Slack complaints flat has been tuned to the test.

For a non-design function, substitute the nouns. The Slack agent is your contract-drafting bot, your vendor-onboarding assistant, your monthly close summarizer. The weekly complaint aggregation is the review meeting you already hold. The harness is the part you probably have not built: a fixed set of realistic requests, run with and without the policy, counted by a machine.

Why this pattern travels

The reason the Vercel loop generalizes is that it never treats the policy document as the product. The product is the split between what a machine can verify and what a human has to judge, and the loop’s whole job is to push the boundary between those two, one correction at a time, with a comparison run after every push. Substrate that adapts to agents was our name for tokens and constraints that move with the agent instead of behind it. The Vercel loop is that idea with a scoreboard.

The failure mode it protects against is the one every agent governance program eventually meets: the policy file grows, the confidence in it grows faster, and the human review step gets shortened on the theory that the file now covers it. Vercel’s own six pages say the file did not cover it. 57% fewer failures, zero pages shippable. Both numbers are true, and the second one governs the review process.

Do this now

Pick one agent your company runs in a non-code function and one policy document it reads. Then build the smallest version of the Vercel loop.

Write down five realistic requests that agent handles. Take the messy ones from last month, the requests that already caused a complaint. Run each one with the policy document loaded and once without. Have a machine count what it can count: missing sections, wrong formats, violated constraints you can express as a check. Have a human mark each output as shippable or blocked.

You will get two numbers. The first is how much the policy document reduces countable failures. The second is how many outputs still need a person. Publish both to the team that owns the agent, and institute the Vercel rule: from now on, no one fixes an output. They fix the guidance, the reusable mechanic or the check, and rerun the five requests.

The first number tells you whether the file is worth maintaining. The second tells you where the human review step still lives. Do not let anyone remove that step until the second number goes to zero on its own.

---

This analysis synthesizes How our agents build on-brand pages with design.md (Vercel, August 2026).

Victorino Group helps engineering and operations teams build eval loops around the policy files their agents read, so review effort moves to the failures only a human can judge. Let’s talk.