Labs and Lawmakers Back Independent AI Evaluators. No One Has Settled Who Pays Them.

“There is also no settled system for funding independent evaluation.” Anthropic made that admission in a statement CNBC quoted, which also conceded “there are, as yet, no standards for what information embedded evaluators should have access to, or how they should report what they find.”

Read the admission next to the political support for evaluators and the shape of the problem is clear. According to Ashley Capoot’s CNBC reporting, executives from Anthropic, Google, Meta, OpenAI, SpaceX and Nvidia signed a one-page White House accord described as morally binding. CNBC also reports that Dario Amodei pledged to embed independent evaluators at Anthropic “as part of a voluntary accord he presented in late September.” The FRONTIER Act, introduced in July, includes independent verification organizations. California signed two related bills. Connecticut and Virginia have taken steps.

The principle has sponsors in labs, the White House, a bill in Congress and several states. The plumbing is unsettled. In Anthropic’s words, there is no standard yet for what embedded evaluators may see or how they report, and no settled system to pay for the work.

A disclosure before going further: Victorino Group sells independent audits of AI systems. We have a commercial interest in this question being answered well, and the test we propose at the end applies to us as much as to anyone else.

The money is arriving faster than the rules

The evaluator market is growing on capital that the CNBC piece describes in some detail.

METR announced in August that it had raised commitments of around $71 million over the last six months. Its IRS filing puts total 2024 contributions at $13.6 million. Those two figures measure different things (commitments over a window versus contributions received in a year), so they should not be read as a growth multiple. METR employs fewer than 50 full-time staffers.

Vals AI says it has grown from eight employees to roughly 30 this year, and it announced a $40 million funding round in August.

Anthropic, for its part, will fund embedded evaluators from Faculty (Accenture) directly. Its stated long-term view: “Long-term, we think funding should come from pooled or government sources.”

Anthropic’s own terms for those evaluators are specific. CNBC reports Dario Amodei describing permissions “mostly comparable” to internal teams and “the right to publish key findings”, with Anthropic keeping “the narrow ability” to redact. Those terms apply to one lab only.

Anthropic’s long-term funding sentence deserves attention. In our reading, the lab paying for its own evaluators is saying, on the record, that this arrangement should not be permanent. It is a candid position. In our reading, the current arrangement carries the conflict financial audit was built to manage.

Financial audit already ran this experiment

An auditor paid by the company it audits has an obvious incentive problem. Accounting did not solve it by assuming good character. It built structure around the fee: independence rules, disclosure of what the auditor was paid and for what, rights of access to books and records, and a signed public opinion the client does not get to edit.

None of that structure guarantees honesty. It makes dishonesty more expensive and more visible.

AI evaluation has the fee and very little of the structure. Suresh Venkatasubramanian put the requirement plainly in the CNBC piece: “If you want true third-party evaluation, you need true independence financially and otherwise.”

In our reading, the “otherwise” covers access and reporting. Anthropic’s statement says no industry standard yet covers either. An evaluator who could only see what the lab chose to show, and could only publish what the lab agreed to release, would be producing something closer to a sponsored review than an audit. The label on the report does not change that.

We have written before about self-reported scores diverging from independent runs and about buyers writing the certification standard themselves. This piece adds the funding question.

When the rules for talking to evaluators are unwritten

A letter titled “OpenAI cannot make AI safe on its own”, signed by Tomek Korbak, Jasmine Wang and Mikita Balesni, adds a second dimension. All three were fired by OpenAI. Our sources place the letter in October 2026, though the page itself carries no date.

The circumstances are disputed, and the letter is one side of the dispute. CNBC reports that two of the three researchers say they believe they were dismissed over how they communicated with evaluators. OpenAI cites “violating our policies on accessing and handling sensitive company information.” We cannot adjudicate between those accounts and will not try.

What the letter recommends is useful regardless of who is right about the firings. The authors ask OpenAI to keep embedded third-party auditors and to publish clear rules for work with outside safety organizations. OpenAI, for its part, said it is “actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks.” The authors’ reasoning: “If conduct that was considered normal last month now constitutes grounds for sudden dismissal, everyone at OpenAI is left guessing where the line is.”

That is an access problem seen from inside the lab. If the rules for talking to an evaluator are unwritten, then the evaluator’s access depends on whatever the lab’s employees judge to be safe on a given day. In our view, an evaluator cannot be fully independent of an organization whose staff are unsure whether speaking to it is allowed.

Even Microsoft’s CEO is asking for auditability

Satya Nadella, in an X post that CNBC’s Greg Iacurci reported, listed “independent controls and auditability” among six principles for AI and said treating frontier models like insider risks “is a way to build such a system.”

In our view, treating a model as an insider risk is a useful frame for anyone procuring evaluation. As we read the idea, an insider risk program assumes the subject has legitimate access and may misuse it, and it depends on controls the subject does not own. An evaluation regime that the evaluated party funds, scopes and approves for release would fail that test on every count.

The three questions to put to any evaluator

Independence is a property you can check before signing. Before you rely on a third-party evaluation of a model, an agent or a vendor’s AI system, get written answers to these.

Who pays, and how is the fee set? If the evaluated company pays, ask whether the fee depends on the outcome, on repeat business, or on scope the company controls. Pooled or public funding, which Anthropic itself names as the long-term target, would reduce this concern. Direct funding keeps it, and the buyer should know which arrangement applies.

What can the evaluator see? Ask for the access terms in writing. Model weights, training data summaries, system prompts, internal incident logs, staff interviews: which of these were available, and which were refused? Since there is, in Anthropic’s words, no standard yet for what embedded evaluators should have access to, every engagement sets its own terms. Those terms are the scope of the assurance you are buying.

Can the evaluator publish without approval? A finding that requires the evaluated party’s consent to release is a finding the evaluated party can suppress. Ask whether the evaluator retains the right to publish a summary, and whether any prior engagement ended in a report that was never released.

If an evaluator cannot answer all three, the report it produces tells you what the lab permitted someone to check. That still has value. It should not be priced or relied on as independent assurance.

For buyers who want evidence the lab cannot shape at all, the complement is evaluation you run yourself, with held-out honeypots now, or, as the tooling matures, double-blind setups the vendor never sees in advance.

Do this now

Pull the last third-party AI evaluation your organization relied on, whether for a model selection, a vendor review or a board report. Write down who paid for it, what access the evaluator had, and whether the evaluator could have published a negative finding without permission. If any answer is “unknown”, request it from the vendor before the next renewal.

Then apply the same three questions to every assurance provider you use, including us.

---

This analysis synthesizes AI’s quiet safety gatekeepers are stepping into the spotlight (CNBC, October 2026), OpenAI cannot make AI safe on its own (Tomek Korbak, Jasmine Wang, Mikita Balesni, October 2026), and Microsoft’s Nadella says AI needs an ‘emergency brake’ that humans control (CNBC, October 2026).

Victorino Group runs independent AI audits. Ask us the same three questions. Let’s talk.