Public Benchmarks Said 90%. On Four Real Warehouses, the LLM Scored Zero.

A pure large language model, pointed at four real enterprise data warehouses and asked to answer questions that real business users had actually asked, produced an accuracy score of zero.

That number comes from Beaver, a text-to-SQL benchmark built by Michael Stonebraker, a professor at MIT and the 2014 recipient of the ACM A.M. Turing Award, together with Peter Baile Chen, a PhD student at MIT CSAIL. On the public leaderboards that vendors quote in sales decks, the same task looks solved. Spider 1.0 sits in the 80-plus percent range. BIRD-SQL reports above 90 percent. Agentic solutions on Spider 2.0 also clear 90 percent.

A ninety-point spread runs far past anything rounding or tuning would explain. It signals that the public benchmarks measure a task that does not exist inside your company.

How Beaver Was Built

The methodology is the interesting part, because any enterprise can reproduce it in a week.

Stonebraker and Chen started with MIT’s own administrative data warehouse: an Oracle system with more than 1,400 tables. They pulled real queries out of the production query log. Then they had real business users write the natural language question that each query answered. The result is a set of (natural language, gold SQL) pairs where both halves came from the organization rather than from a benchmark author’s imagination. They repeated the exercise across three additional warehouses.

Nothing in that procedure requires a research lab. You have a query log. You have business users. You have a warehouse whose schema nobody fully understands. The benchmark you can build from those three assets is worth more than every public leaderboard combined, because it is the only one that measures your warehouse.

The Accuracy Ladder

The paper reports a progression, and the shape of it matters more than any single figure.

A pure LLM scored zero. Adding retrieval-augmented generation, prompt engineering, and agentic scaffolding lifted accuracy into the 10-plus percent range. Then the authors did something no production system can do: they handed the model the from-clause tables and the actual join clauses taken from the gold SQL. With the hardest part of the problem solved for it in advance, the model reached the 30-plus percent range.

Their own framing of that ceiling is worth quoting: “Note that this is 50+% lower in accuracy that the public benchmarks and is the difference between the technology has promise and the technology does not work.” (The typo is in the original.)

Read the ladder as a diagnostic. Zero to ten is what tooling buys you. Ten to thirty is what schema knowledge buys you, and it was smuggled in from the answer key. Nobody has yet shown what closes thirty to ninety on a real warehouse.

Why Real Warehouses Break the Model

Four properties of production data stores explain the collapse, and each one is a governance artifact rather than a modeling problem.

Schema rot. The authors give a concrete example: there may be six different columns across various tables all labelled salary, with overlapping and undocumented semantics. One might be net after taxes and commissions. Another might be gross. A human analyst resolves this by asking a colleague who has been there eight years. A model resolves it by guessing. Schema changes land multiple times a year, so any documentation that did exist is stale by design.

Idiosyncratic data. MIT’s calendar includes a “J-term.” Its buildings are numbered rather than named, so the Stata Center is officially Building 32. No amount of general world knowledge recovers that mapping. Every company has its own version: a product code that means something only to finance, a status flag whose fourth value was added for one 2019 migration and never removed.

Query complexity. Real queries “rarely have just one join and often two or three.” Benchmark queries skew simpler. Join selection is exactly where the model was shown to need the answer key.

Structural contamination. This is the part that should reframe the whole procurement conversation. As the authors put it: “It is widely known that an LLM can only find data it has seen before. Almost all data warehouses we know about are behind serious enterprise access controls and are not publicly available.”

The reflexivity there is uncomfortable. The access governance you built to protect the warehouse is the same reason no frontier model was ever trained on anything resembling it. The better your controls, the further your schema sits from the training distribution, and the worse a general-purpose model performs against it. Governance maturity and out-of-the-box text-to-SQL accuracy move in opposite directions.

We have argued before that agents inherit the data layer your humans were quietly absorbing, and that the semantic layer stopped being a BI topic. Both arguments were qualitative. Beaver supplies the quantity: the cost of undocumented semantics, expressed as accuracy, is roughly sixty percentage points.

The Silent Failure Mode

A code-generation agent that gets it wrong usually announces itself. Tests go red, the build breaks, the linter complains.

A text-to-SQL agent that gets it wrong returns a number. The number has the right type, plausible magnitude, and a chart wrapped around it. It flows into a board deck. Nobody sees a stack trace, because there is no error: the SQL was valid, it just joined the wrong salary column.

That is why the accuracy figure is a governance number rather than a product-quality number. At 30 percent accuracy with no error signal, roughly two of every three answers reaching a decision-maker are wrong and indistinguishable from the correct third. This is the same failure shape we described in the domain competence wall: the model’s output is fluent in a domain where fluency and correctness are uncorrelated. And it is why artifact metrics get gamed while outcome tests hold applies directly here. A leaderboard score is an artifact metric. Your query log is the outcome test.

One disclosure before the recommendation: Stonebraker and Chen built both Beaver and Rubicon, the alternative system they go on to recommend. They have a stake in the conclusion that pure LLMs fail. The methodology is reproducible independently, which is what makes the finding usable regardless of where their commercial interest sits. Verify it on your own warehouse and the authors’ incentives stop mattering.

Do This Now

Build your own Beaver. Two weeks, one analyst, no vendor involved.

Pull the 100 most frequently executed queries from your warehouse’s production query log, weighted toward anything that feeds a recurring report.
Have the business users who own those reports write, in plain language, the question each query answers. Do not let an engineer write these; the phrasing is the test.
Freeze that set as (question, gold SQL) pairs. This is now your private benchmark, and by construction it cannot leak into anyone’s training data.
Run every text-to-SQL vendor and internal prototype against it. Score execution-result equivalence, not SQL string similarity.
Put the resulting number in the procurement document. Make the vendor’s published leaderboard score irrelevant to the decision.

The authors’ procurement rule is blunter than anything we would write ourselves: “any text-to-SQL benchmark should address the difficulties of real-world data stores. Those that do not are of academic interest only and do not reflect reality.”

There is a second lever hiding in the ladder. Ten to thirty percent came from resolving schema semantics. That means schema documentation now has a measurable price attached, expressed in accuracy points on a benchmark you control. Every year that six columns named salary stay undocumented has a number next to it. Take that number to whoever declines to fund data governance work.

This analysis synthesizes If You Think You Can Do Real-World Text-to-SQL (BLOG@CACM, Michael Stonebraker of MIT and Peter Baile Chen of MIT CSAIL, 2026) and the public Beaver benchmark and leaderboard.

Victorino Group helps data and engineering teams build private benchmarks from their own query logs before signing an AI analytics contract. Let’s talk.