Precision Before Recall: Figma Shipped the Ordering Nobody Guesses

Week one of Figma’s security review agent produced 27 findings. Four of them were real. “Only about 15% of findings (4 of 27) were valid,” the security team writes, and that sentence is the whole rollout problem in one line. A reviewer at 15% precision does not have a tuning issue. It has a trust problem, and every bad comment it posts spends someone else’s attention.

The instinct at that moment is to reach for the bug history. Every mature engineering organization has one: bug bounty reports, incident write-ups, audit findings, the corpus that tells you whether your agent catches what your last three years of attackers found. Figma built exactly that corpus, 66 tasks in total, and then said the thing worth quoting: “the historical bugs you already have can only measure recall; they barely help with the precision you must fix first.”

That is not a limitation of Figma’s corpus. It is arithmetic. A corpus of known-real bugs cannot produce a false positive, because every item in it is true. Run your agent against it and you learn what fraction it finds. You learn nothing about what fraction of its output is noise, and noise is what determines whether developers keep reading.

Precision Is a Production-Only Measurement

Figma’s ordering follows from that. Shadow mode first, with findings visible to the security team and nobody else. Then hand-label the false positives, which is the step no benchmark performs for you, because a false positive is defined against your codebase and your threat model rather than against a general notion of vulnerability. Then write the policy. Then release developer-facing PR comments, gated: “We held back developer-facing PR comments until precision stayed above 70% over a two-week lookback, with no embarrassingly bad false positives.” Precision reached 80% within a month. Only after that did review become a merge requirement.

Four things in that gate are worth copying verbatim. It names a number. It names a window, which stops a lucky week from clearing the bar. It adds a qualitative veto for the finding that is technically borderline and socially catastrophic. And it attaches the gate to a specific surface, developer-facing comments, rather than to the agent as a whole.

That last piece is what most rollouts flatten. Figma runs asymmetric bars by surface: hooks that steer the agent “don’t carry a high precision bar, because they’re mostly invisible to the human author.” A wrong hook costs compute. A wrong PR comment costs credibility. Same model, same findings, different thresholds, because the currency being spent is different. One guidance hook cut a logging-safety bug class by about 50%.

The Policy Is Ninety-Nine Lines of Precedent

The artifact that moved precision is a document, not a model upgrade. “Ninety-nine lines, 2,560 words, and 68 precedents.” The word doing the work there is precedents.

The tempting format is rules. Do not flag SQL injection in dbops. That kind of line makes the finding disappear and teaches the reviewer nothing, so the next variant of the same pattern comes back. Figma writes the reason instead: “dbops is only run by highly privileged operators who already have direct database access.” A reviewer that knows why generalizes to the adjacent case. A reviewer that knows only the exception does not.

And the policy’s contribution is falsifiable, which is rarer than it should be in write-ups like this. With the policy ablated to nothing, the same reviewer cleared 44.4% of the corpus on payout weight. With the policy restored, 64.2%. That is a measured delta on a document that most teams would classify as overhead. “The policy is the threat model,” Figma writes, and the ablation is the receipt.

Two models run in parallel by design, Claude Code with Opus 4.8 at xhigh effort and Codex with GPT-5.6 Sol at high effort, “because they miss different bugs. If either model surfaces a finding, we bubble it up.” Union, not intersection, which is the correct choice when recall is the scarce quantity and a policy is already suppressing the noise. Cost stays unremarkable: “a pull request review runs about $0.50 median spend, rarely more, since most PRs have nothing to flag.” The first full monorepo sweep surfaced “more than a hundred latent vulnerabilities, including two criticals missed by traditional SAST tools, which we patched immediately.”

Figma is also honest about what the corpus proves. The union catch rate is 75.8%, and they refuse to round it up into coverage: “a 75.8% union catch rate is 75.8% of the bugs that got all the way past our existing controls.” The corpus is “a floor for the agent to clear rather than proof of coverage.” Of the 66 tasks, 46 came from their HackerOne programme, 24 of those from a single top researcher, and 20 from internal incidents and audits. A benchmark shaped by one researcher’s taste measures that taste as much as it measures the agent.

Vercel Weights Recall Twice, and Is Right To

Published the same month, Vercel’s DeepsecBench scores models with Score = 100 x 5PR/(4P+R), weighting recall twice as heavily as precision “because missed vulnerabilities will go unfixed, while false positives don’t make your codebase less secure.”

Read alongside Figma, that looks like a direct contradiction. It is not, and the reason resolves both. A benchmark’s false positive is paid for in compute and a grader’s time. A PR comment’s false positive is paid for in a developer’s trust, and trust does not refill on the next run. Vercel is measuring model capability, where under-detection is the failure that matters. Figma is deploying into a human workflow, where over-detection is the failure that ends the programme. The variable that decides the ordering is whether a person’s attention is being spent on each finding.

The benchmark’s own numbers argue for its weighting. Scope is 50 entry-point files against “a golden set of 231 human-judged findings”, run three times with the median published. The difficulty floor is brutal: “the best run finds 30.7%, and 20 of the 25 runs come in under 20%.” When the ceiling is under a third of known findings, precision is not the binding constraint on the model side. Contamination is handled by withholding everything: “We don’t disclose the repository, the commit, the files, or the findings, so there is nothing for models to train against.” A full production pass extrapolates to “roughly $1,200 for a Kimi K3 sweep, or over $5,000 for the top-scoring frontier model from OpenAI.”

One supply-side detail belongs in any procurement conversation: “Anthropic’s most capable model, Fable 5, is absent because it declines security work, including defensive tasks.” Model availability for defensive security is a vendor policy variable, not a capability one, and it constrains architecture. Vercel’s closing asymmetry is the reason to care at all: “a model that can read the source finds what an attacker can only probe for. The advantage is real, but it only counts if you use it first.”

Do This Before You Post a Single Comment

If you are standing up a security review agent this quarter, run it in shadow mode and instrument one metric: the share of findings your security engineers label real, over a rolling two-week window. Publish that number internally before you publish a single comment to a developer. Then write the policy as precedents with reasons attached, and prove its value by ablating it, the way Figma did with 44.4% against 64.2%.

Set the precision bar per surface, not per agent. Invisible surfaces get a low bar and fast iteration. Human-facing surfaces get a named threshold, a lookback window, and a veto for the embarrassing finding. Make merge-blocking the last step rather than the launch, for the reason Figma states plainly: “Making review a merge requirement turned it from a nice-to-have into infrastructure, and infrastructure has to be boring to be trusted.”

Your bug history still matters. It sets the floor. It just cannot tell you when you have earned the right to interrupt a developer.

We have written before about the governance deficit in AI code review, about what Bugbot revealed regarding developer experience, about detection outpacing remediation, and about governance packaged as product in restricted cyber models. This is the first piece where someone published the threshold.

---

This analysis synthesizes How Figma stays ahead of vulnerabilities with agents (Figma, July 2026), DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilities (Vercel, July 2026).

Victorino Group helps engineering organizations design the rollout ladder for security review agents, from shadow mode to merge-blocking, with named precision gates per surface. Let’s talk.