Synthetic Respondents Need a Label Before They Need a Benchmark

Aaru was valued at $1 billion as of September 2026. Simile raised $200 million at a $2 billion valuation in July 2026. Both companies work in the same category: survey answers generated by language models that stand in for human respondents. Contrary Research reviewed the evidence behind that category in a report updated on September 22, 2026, and its reading is uneven. Synthetic panels fit reasonably well on aggregates. On subgroups and on individuals, the fit is poor.

Two disclosures before the numbers. Contrary is an SEC-registered adviser, and its report states that its clients “may take any position” in the companies it names. The Verasight, AAPOR and Nature figures below also reach us through Contrary’s summary, and we have not read those primary documents. Treat every figure in this piece as Contrary’s account of it.

What the tests found

Contrary reports a Verasight test on a zoning measure. The synthetic panel predicted 59% support. The real figure was 28%. The synthetic panel also produced 0% “don’t know” answers, against 29% among real respondents.

That zero deserves more attention than the headline miss. Roughly three in ten real people had no settled view on the question. The simulated respondents all had one. Any team using that output to size demand, plan a campaign or read public sentiment would have seen certainty where the population had none.

Verasight’s testing, as Contrary summarizes it, also measured error by subgroup. It averaged 8%. For Black respondents it rose to 15%, and for respondents of other races it reached 20%. The error concentrates in the groups a decision about product, credit or care often needs to read on their own.

Contrary also cites a Nature study from 2026 that found “the model consistently overestimates how large effects are.” If that finding holds beyond its setting, a team comparing two messages or two offers on a synthetic panel would see a wider difference between them than real respondents would produce.

The profession already wrote the rule

The survey research field has not waited for the technology to mature. Contrary quotes an AAPOR task force on two points. First, “a margin of error cannot be produced from synthetic responses.” Second, “researchers should be transparent about and clearly distinguish between data derived from human respondents and data generated by AI systems.”

Gallup, per the same report, “has ruled out using simulated responses for published estimates or tracking.” It still validates the method for questionnaire pre-testing and for hard-to-reach populations.

Put those two positions next to each other and a working policy appears. Synthetic respondents can help design a survey, test wording and explore a population that is hard to reach. They cannot carry a published estimate, and whatever they produce must be marked as generated. Both parts of that policy govern disclosure and scope of use, and both can be enforced today.

The Axios case

Contrary describes one instance of the rule failing in public. Axios “reported a finding from Aaru that “a majority of people trust their own doctors and nurses,” without noting anywhere that those respondents were models rather than patients.”

We have only Contrary’s description of that story, and we have not seen a response from Axios or Aaru to it. Taken as described, the case is instructive for what it lacked. The finding concerned trust in healthcare, a subject where readers, hospital administrators and insurers all have reasons to act on survey results. A reader of that story had no way to know the “people” in it were simulated.

My reading is that a more accurate model would not have repaired that story. A better model would have produced a better number, and the number would still have arrived without the one sentence that told the reader how it was made. The benchmark question (how close is synthetic data to human data?) matters. The labelling question comes first, because without a label nobody downstream can even ask the benchmark question.

Why this reaches the enterprise

A synthetic result does not have to stay where it was produced. It can move from a vendor dashboard into a research summary, from the summary into a slide, and from the slide into a decision about pricing, a financial product or a patient program. Every hop is a chance to lose the note that says how the data was made.

We made the provenance argument for generated media in One Lab Added an Invisible Watermark. The Other Shipped the Delete Button.: when provenance depends on a vendor’s setting, it can change without notice. Research data has the same weakness. A table of percentages looks identical whether a person or a model answered the questions.

Three kinds of decision carry this exposure most directly: marketing, finance and healthcare. In marketing, an overstated effect size can justify a campaign that underperforms. In finance, subgroup errors like the ones Verasight measured, if they carried over, would fall on the kind of groups fair lending reviews exist to protect. In healthcare, a trust or adoption figure can shape how a program is designed for patients who were never asked. For operational context on marketing teams running automated systems, see also Marketing Just Got Its First Engineering Outage.

Synthetic panels can stay in the toolkit as long as every consumer of the output knows which rows came from people.

Do this now

Add a provenance field to every research dataset your organization buys or builds. Each record carries one value: human, synthetic, or blended. Reports and dashboards read that field and print it next to every figure derived from it. This one change makes the rest of the list enforceable.

Then set four rules:

No margin of error on synthetic figures. AAPOR’s position is explicit. If a vendor report shows one, ask how it was computed.
Synthetic output stays internal until humans confirm it. Follow Gallup’s line: pre-testing, wording checks and hard-to-reach exploration are fair uses. Published estimates and tracking series are not.
No synthetic subgroup cuts in decisions. With reported subgroup error of 15% to 20% for some groups, any segment-level conclusion needs a human sample behind it.
Ask vendors for their “don’t know” rate. A panel that never hesitates is telling you something about its calibration. Request subgroup-level error from their validation work as well.

The valuations show that investors want this category. The profession has already told them what the output can and cannot carry. What remains is making sure the label survives the trip from the vendor to the decision.

This analysis synthesizes AI Polling: How Synthetic Surveys Could Predict Real Behavior (Contrary Research, September 2026).

Victorino Group helps research, marketing and data teams put provenance controls on AI-generated data before it reaches a decision. Let’s talk.