Shopify Named the Escape Hatch UNSAFE_ and Went from 50% to 98%

A few weeks after Shopify promoted a new mobile end-to-end testing API into blocking CI, the Shopify app suite reported 98% test stability, measured as individual test successes divided by total runs. The old API sat at 50%.

The number before the rewrite is the more useful one. At 50%, the suite “had gotten to the point where it was blocking more good PRs than bad ones,” Michael Garfinkle writes, and “it got so bad that we had to pull the E2E suite from our PR checks entirely.” A test suite outside CI is a test suite nobody is accountable for.

What makes the write-up worth reading twice is the diagnosis that preceded the rewrite. Garfinkle’s team had been treating flakiness as a maintenance problem: bail out the water, fix the top offenders, repeat. His conclusion after that cycle: “The problem wasn’t that we were bad at bailing out the suite; it was that the framework itself kept creating the same failures. No amount of cleanup would solve that. We had to fix the underlying system.”

The framework was manufacturing the failures. So they changed the framework, and the design choices they made generalize well past mobile testing.

Mechanic One: Make the Unsafe Path Expensive to Name

Guardrails are easy to build as prohibitions. The bad option is removed, and engineers who genuinely need it route around the framework entirely, which puts the workaround somewhere your tooling cannot see.

Shopify took the other route. The escape hatches exist and are named to be embarrassing: “Escape hatches are prefixed UNSAFE_. Options exist that bypass the guardrails (like custom timeouts or script injection), but they’re named to discourage reaching for them. UNSAFE_timeoutInSeconds in a test is a signal for review.”

Three properties fall out of that one convention. The bypass stays inside the framework rather than pushing the engineer out of it entirely. The bypass is greppable, so the count of UNSAFE_ call sites is a real metric that a team can watch move. And the bypass is legible in a diff, so review catches it without a linter, a policy document, or a meeting.

This is the part that transfers directly. If your repo has a dangerous operation that is occasionally legitimate, do not remove it and do not leave it named neutrally. Rename it so that using it is a public act. force_delete reads as an option. UNSAFE_force_delete_bypasses_audit reads as something you will have to defend.

Mechanic Two: Validate the Validator

The second design choice attacks the failure mode that makes E2E suites untrustworthy even when green. In Shopify’s new API, “every step carries an assertion. You can’t tap, wait, or type without declaring what the screen should show afterward. If the app leaves the expected state, the test fails at the step where reality diverged, not four actions later when something downstream breaks.”

Step-level assertions alone would not be enough, because an assertion that is always true asserts nothing. So the API adds a rule that governs the assertions themselves, stated verbatim in the article: “an assertion must be false before the action, and true afterwards.”

That rule is the interesting one. It puts the tautological check on the wrong side of a rule. If a developer asserts something that was already true before the tap, the assertion violates the rule and the test fails, rather than entering the suite to pass forever without exercising anything. The framework is not checking the app here. It is checking the check.

Anyone who has audited a green test suite and found part of it asserting on constants knows how much silent debt that rule pays down. The same shape applies far outside testing. A monitoring alert that never fires, a policy engine that never denies, a review agent that never blocks: each is an assertion nobody has proven can be false.

Mechanic Three: Gate Promotion on Repeated Runs

A single green run proves the test can pass. It does not prove the test will keep passing. Shopify put a pipeline between the two: “Before a new test is allowed into the blocking suite, a dedicated pipeline runs it multiple times and rejects it if it fails above a set threshold.”

The article does not disclose the threshold value, and picking one for your own repo is empirical work rather than a number to copy. The structure is what matters. Entry into the blocking suite is a promotion with a demonstrated stability requirement, not a default that new tests inherit by existing. That inverts the dynamic this replaces, where the suite degrades quietly because every new test is admitted on one green run and removed only after it has wasted enough engineer hours to justify the argument.

Why This Reads as an Agent Story

Garfinkle lists an unusual design goal alongside the human ones: the API should be “readable enough for AI agents to write. The surface area is small and the grammar is predictable, which means both humans and AI tools produce correct tests on the first try more often.”

That sentence is the reason this piece belongs next to our work on governing the harness rather than the model. Correctness here comes from shrinking the grammar until the bad construct is hard to express. Better prompting and harder review both act after the fact; the grammar acts before it. An agent writing against an API where every step demands an assertion and where the escape hatch is named UNSAFE_ produces auditable output by default, because the API left it very little room to produce anything else.

Detail’s Towards self-driving codebases argues this thesis directly, that codebases should be made legible to the agents working in them rather than the reverse. Two caveats belong on that citation. The post carries no statistics of its own, no visible byline, and no publication date on the page. And Detail sells tooling in exactly the category it recommends, which makes it a framing to borrow rather than evidence to lean on. Shopify supplies the evidence: a real API rewrite, a stability number before and after, and a suite that went back into blocking CI.

The rest of the Shopify stack is worth noting for how ordinary it is. PaddleOCR reads on-screen text. OpenCV matches icons using grayscale plus color-inverted variants across multiple sizes, with adjacency used to disambiguate duplicate elements. The stack Shopify describes is off-the-shelf OCR and template matching, with no custom model in it. The constraint that produced the number lives in the API surface, which is the same argument we made about the eval environment being production.

Do This Now

Open the API your agents and your junior engineers write against most often, and answer two questions with a grep rather than an opinion.

First: what is the most dangerous thing that API lets a caller do, and what is it named? If the dangerous option is named as neutrally as the safe one, rename it this week. Prefix it, make it long, make it appear in review. You are not blocking the operation. You are pricing it.

Second: which of your automated checks has never failed? Pull the history for your assertions, your alerts, and your policy gates, and list every one that has been green since it was written. Each of those is unvalidated until you can show it going red. Shopify’s rule gives you the test: force the precondition to be false, and confirm the check notices.

Both exercises take an afternoon. Neither requires a framework rewrite, which is the expensive version of this lesson that Shopify paid for and published. Their 50% is the price of finding out late that the framework, not the tests, was producing the failures.

---

This analysis synthesizes How we raised mobile end-to-end test stability to 98% (Shopify Engineering, Michael Garfinkle, August 2026) and Towards self-driving codebases (Detail, company blog, 2026, date not on page).

Victorino Group helps engineering teams redesign the API surfaces their agents write against, so correct output is the cheapest path. Let’s talk.