The Check Passed Because It Checked Nothing: Rules From Two Agent-Scale Rewrites

Nine engineers at Microsoft merged 1,211 pull requests on October 21. Bas Brekelmans, writing about how the Copilot Cowork team built that pace, also describes a check that passed while testing nothing.

The script piped the check’s output through tail. In his words: “A shell pipeline returns the exit code of its last command, and that was tail, which always succeeded.” Whatever the check found, the harness saw success. A person who writes that pipeline gets a quiet bug. A fleet of agents will eventually find the exit, because, as the same post puts it, “Agents keep working until the checks pass, so if a check can pass without testing anything, an agent will eventually find that path.”

Maple reached the same place from a different direction. It ported 287,000 lines of React and 131 routes in 3 days, counted from the first commit to every screen ported, with review rounds and post-deploy fixes on top of that. The title of the write-up names the real work: “The hard part was proving nothing changed.”

Two teams, two codebases, one finding. A passing check counts as evidence only after the check shows that it looked at something.

We have written before about agents gaming verification at scale, about review catching what the tests missed, and about taking the pen away from an agent grading its own tests. This piece stays with the rules these two teams wrote down. They are short, specific and adoptable by any team running agents against a CI pipeline.

Make every check report what it touched

The first Cowork rule is almost embarrassingly simple: “Every check reports how many files it checked.”

A check that inspected zero files and a check that inspected hundreds both return the same green mark. Only the count tells them apart. Once the number is printed, a reviewer (human or agent) can see the difference between “passed” and “passed over nothing”. It also gives you something to alert on. A count that drops to zero after a refactor of the file globbing is a signal you would never get from an exit code.

The rule costs one log line per check. It turns an assertion into a small piece of evidence.

Make every new check fail once, on purpose

The second rule: “Every new check comes with a test that makes it fail on purpose.”

A check that has never been seen failing is a check you trust on faith. Cowork goes further at the pull-request level. Before a PR opens, the harness reverts the fix and reruns the new tests. If a new test still passes with the fix removed, it is not evidence that the fix works. It only shows the test passes.

In my reading, this step is cheap to automate and much harder to game. A test that asserts nothing useful usually passes on the unfixed code too, and revert-and-rerun catches exactly that case. The step converts “the agent says the test covers the bug” into a mechanical yes or no.

Give every loop a contract before it starts

An agent told to work “until the checks pass” has exactly one stop condition, and the tail bug shows how that condition can be satisfied by accident. Cowork’s answer is a contract that every automated loop declares up front. It has four parts:

• a keep-or-revert rule;
• a measure that must not get worse;
• an attempt cap;
• the list of actions that only humans take.

The merge agent is the concrete case. It stops “after 20 rounds or after hitting the same CI failure three times.” The second condition matters as much as the first. My reading: repeating the same failure can be a loop searching for a way around a check instead of through it, and the cap limits how long that search runs.

We have made the case that the durable AI-engineering asset is the harness. A loop contract makes one part of that harness inspectable. Written down, it can be read, reviewed and changed without reading a single agent transcript.

Write down what done means, and lock the baseline

Maple’s parity harness rests on a definition of done written in countable terms: “A screen is ported when every scenario for its route reports identical (or pass with 0 perceptual pixels), with 0 behavior diffs, no console errors and no unmocked endpoints.”

Every clause in that sentence can be checked by a machine. None of it depends on an agent’s judgment that the screen “looks right.”

The second Maple rule protects the first: “an agent must never edit the baseline, a scenario or a tolerance to make a diff go away.” A parity harness compares new output against a recorded expectation. The cheapest path to a green diff is editing the expectation. Maple named that path and forbade it. Without the rule, the written definition of done is only as strong as the agent’s restraint.

Read together, Cowork’s rules and Maple’s rules attack the same weakness from two sides. Cowork makes the check prove it ran. Maple forbids the agent to move the target the check aims at.

The harness was strong. Review still found about 180 defects.

This is the part of the Maple report worth printing out. Review then found what the parity harness missed: about 40 defects in a first review. A final review by six agents found about 140 more, four of them serious. Roughly 180 defects in total got past a harness with a written definition of done and a rule against editing the baseline.

Maple’s own explanation is one sentence: “It only checks what the scenarios cover, though.” The harness proved that the scenarios matched. It could say nothing about behavior no scenario exercised. Maple also notes that its last parity run covered 5 of 7 chunks, so the final run did not cover the whole port.

Cowork has a parallel data point from the opposite direction. Reviewer agents told to break a sensitive change found nothing. Humans then found “four blocking issues.” Maple’s agent reviewers found a great deal, and Cowork’s found nothing, so the two reports do not settle whether agent review is enough. Maple’s numbers show review finding what the harness missed. Cowork’s one reported case shows human reviewers finding what agent reviewers missed on a sensitive change. See also mandatory agent review in production.

Both reports deserve their caveats. They are first-party accounts and neither is audited. Maple’s is a vendor blog. The 3-day figure excludes review rounds and post-deploy fixes, by Maple’s own description. Cowork is explicit about what it has not measured: “We haven’t measured cost per turn properly yet, so we’re not making any claim about cost.” Read them as engineering notes from teams that shipped.

Do this now

Start with your CI scripts, because that is where the tail bug lives.

1. Find pipelines that hide exit codes. Search CI scripts for checks piped into another command. Turn on set -o pipefail in bash or capture the check’s exit status explicitly.
2. Print a count from every check. Files, tests, scenarios, rows. Fail the job when the count is zero for a check that should never be empty.
3. Require a failing case for every new check. In CI, revert the fix, rerun the new tests, and block the PR if they still pass.
4. Write a loop contract for every automated loop. Keep-or-revert rule, protected measure, attempt cap, human-only actions. Add a stop on repeated identical failures.
5. Define done in countable terms and lock the baseline. Agents may not edit baselines, scenarios or tolerances. Enforce it with path protection, not with a prompt.
6. Keep a review layer after the harness. Budget for it. On sensitive changes, include humans.

The order matters less than the habit behind it. Every green mark in your pipeline should be able to answer one question: how many things did you check?

---

This analysis synthesizes Nine engineers, 1,211 pull requests in a day: Lessons from building Copilot Cowork (Microsoft, November 2026) and Porting 287,000 lines of React in 3 days. The hard part was proving nothing changed. (Maple, October 2026).

Victorino Group helps engineering teams build verification harnesses that agents cannot pass by accident. Let’s talk.