Two Frontier Models Scored Zero on the One Metric That Mattered

Fable 5 and Opus 4.8 both scored zero on conversion rate in a website-rebuild benchmark run by AI consultant Vin Vashishta in July 2026. Neither model added conversion tracking to the site it built, unless explicitly told to. Both models had strong scores on the artifacts they produced: clean code, working pages, passable design. On the one number the business actually needed, both failed completely.

Two weeks later and a different domain entirely, Brown University’s ECON 1170 take-home midterm averaged 96 out of 100, with 40 students posting perfect scores. The professor then announced an in-person final covering the same material. Eighteen of those students dropped the course. Nine more no-showed for the exam. Twenty-two of the 27 who vanished had scored perfectly on the take-home. The students who did sit for the in-person final averaged 48.

Different industries, different stakes, same structural failure: the artifact looked finished, and the artifact lied.

What an Artifact Metric Actually Measures

An artifact metric scores the thing produced, not the outcome the thing was supposed to cause. A benchmark score measures whether code compiles, whether a page renders, whether a response matches a rubric. A take-home exam score measures whether an answer sheet was filled in correctly by the time it reached a grader, without measuring who or what filled it in.

Both are proxies. Proxies are useful because they are cheap and fast to check. They are dangerous because cost and speed are exactly what gets optimized once the proxy becomes the target. Vashishta’s website-rebuild benchmark asked models to rebuild a business site. It did not ask them, by default, to verify that the rebuilt site converted visitors into customers. Both models delivered a site. Neither delivered a business result, because nothing in the task forced them to treat the business result as the target.

Brown’s midterm asked students to answer questions at home, unsupervised, with any tool available. It measured whether a correct answer sheet arrived. It did not measure whether the student who submitted the sheet could reproduce that competence under exam conditions. The take-home was graded on the artifact. The in-person final measured the outcome: can this person actually do the thing the artifact claimed they could do.

The Gap Is the Diagnostic

The distance between the artifact score and the outcome score functions as signal, precise and legible. A 96-average take-home next to a 48-average final tells you, precisely, how much of that 96 was earned competence and how much was borrowed. Princeton’s own student survey put a number on the borrowing: 29.9% of respondents admitted to using AI for cheating on at least one exam or assignment. Brown’s provost office found even higher routine use: 56% of undergraduates and 67% of graduate students report using generative AI daily or weekly.

At that scale, the behavior reads closer to standard practice than fringe outlier activity, quietly normalized inside a grading system that never checked for it because it never had to. The take-home format worked fine for decades because the underlying capability to produce a correct answer sheet and the underlying capability to hold that knowledge under pressure were, for most students, the same capability. Generative AI decoupled them. The grading system did not notice because it was still measuring the artifact.

Vashishta’s benchmark makes the enterprise version of the same point, with the caveat that his numbers come from his own test design, not an independently replicated study. He is not claiming one model is smarter than another; the finding holds regardless of which frontier lab produced which score. His claim is narrower and more useful: artifact-level benchmarks, the kind most AI evaluations run today, do not surface business-outcome failures unless the outcome is written into the test. A model can pass every artifact check on a website rebuild and still ship something that converts at zero, because conversion was never the thing being scored.

Why the Fix Is Not “Add a Better Metric”

The instinct after seeing a gap like this is to bolt a new metric onto the existing process: add a conversion-tracking check to the QA pass, add an oral component to the exam. That helps, but it treats the symptom. The deeper problem is sequencing. Both failures happened because the outcome test was designed after the artifact process was already in production, as an afterthought bolted onto something that was built to optimize a different signal.

Brown’s in-person final was not part of the original course design. It was a response to a scandal, added under pressure, months after the take-home format had already shaped how students prepared. It worked as a diagnostic. It arrived too late to be a design principle. Vashishta’s outcome-based benchmark exists because conventional AI benchmarks kept missing business failures that showed up the moment a client asked “did revenue move.” The benchmark had to be built from scratch, outside the frontier labs’ own evaluation suites, because none of those suites had been designed around the outcome from the start.

The governance lesson generalizes past both domains. If you are evaluating an AI system, a vendor, or a team, and the only scores available are artifact scores, treat that as an open question, not a completed evaluation. Ask what the outcome test would look like, whether it currently exists, and who would notice if the artifact and the outcome diverged by 48 points.

Do This Now

Before you sign off on any AI-assisted deliverable, whether it is a codebase, a report, or a hiring assessment, write down the outcome the artifact is supposed to produce, separately from the artifact itself. Design a check for that outcome before the work starts, not after someone gets suspicious. If you cannot articulate what the real-world result should look like independent of the deliverable, you do not yet have a metric. You have a description of the thing you are hoping happens.

---

This analysis synthesizes Fable 5 vs Opus 4.8: Outcomes-Based For Frontier AI Labs (Vin Vashishta / High ROI AI, July 2026), We Cannot Choose to Become Idiots (Ars Technica, July 2026).

Victorino Group helps teams build outcome tests into AI adoption instead of trusting artifact metrics. Let’s talk.