Two Decisions, One Crawler: Training Consent Splits From Search

Fewer than 1% of sites on Cloudflare’s network choose to block Search bots. 17% enable some mechanism to block AI training. Both figures come from Cloudflare’s own network data, published alongside a September 2026 change to how those preferences are expressed.

Those two numbers describe one population of site owners with two very different appetites. Almost nobody wants to disappear from search. A meaningful minority does not want their pages used to train a model. There was no way to hold both positions at once, because the largest operators ran a single crawler that did both jobs. Blocking the training use meant blocking the index.

That is a granularity failure in the enforcement mechanism, and it is worth naming precisely, because the same failure shows up far away from crawlers.

The preference and the control are different objects

Cloudflare’s post is blunt about what a robots.txt file is. In its words: “Anyone can publish [a robots.txt], but it cannot identify who is crawling, determine why they are crawling, or stop a crawler that ignores it.”

A published file states a request. Honouring it is voluntary, and the file has no way to tell one requester from another. Identity and intent both live outside the document. So a site owner who wanted a split decision had nowhere to express it against the largest operators, and a mixed-use crawler that wanted to honour one had nothing to read.

The usual answer was to block. Cloudflare positions against that too: “blocking removes a crawler. It doesn’t change how crawlers behave.” Blocking is a blunt instrument at the perimeter. It removes the traffic and leaves the policy question untouched, which matters when the traffic you are removing is also the traffic that puts you in an index.

One caveat before going further. Cloudflare operates the network that enforces these settings, so it holds first-party authority over how the product behaves and a commercial interest in how the policy is framed. Everything below is from the announcement. No individual byline appears on it.

Three behaviours, four values

The replacement for the single “AI bot” category is a taxonomy of three behaviours, per Cloudflare’s post:

• Search: building an index.
• Training: training or fine-tuning a model.
• Agent: user-directed agents acting on behalf of a human.

One bot can exhibit more than one of these. That is the whole problem restated as a data model, and it is the first version of the problem Cloudflare’s control surface can act on.

Each behaviour then gets one of four values at the domain level: Allow, Disallow AI Training, Block on pages with ads, or Block. The asymmetry is the interesting part. Disallow AI Training exists only for Training. Cloudflare gives no equivalent for Search or Agent.

A site owner can now say: index me, do not train on me, and let user-directed agents through. Three decisions, three answers, one domain.

What the Accountable designation actually certifies

Cloudflare also introduced an Accountable designation for operators that stop forcing the tradeoff. It is a four-requirement rubric:

1. An opt-out mechanism for training, via robots.txt or similar.
2. An opt-out for AI summaries.
3. URL-level visibility into which pages were made available for training, plus metrics on how content appeared in search.
4. Assurance that opting out of training will not affect traditional search results.

Read requirement 3 twice. It is the one that turns a promise into something checkable. A per-URL record of what was made available for training is a disclosure surface, and disclosure is the enforcement that survives when a site owner cannot audit the operator directly.

Now the caveat that governs how much the designation is worth. The rubric explicitly admits time-bound commitments, not only shipped capability. Apple, Google and Microsoft are designated Accountable partly on that basis. Amazon, Anthropic, Meta and OpenAI qualify because they run separate Search and Training crawlers, so the training crawler can be blocked without touching search. Those are two different kinds of evidence sitting under one label, and a buyer should read the label as “obligated” rather than “already built”.

The concrete case: Bing’s domain-level no-training support in robots.txt is targeted for early 2027. Until it ships, selecting Disallow AI Training conveys no no-training preference to Bing. The setting will be selected and mean nothing at that particular destination.

The constraint that reveals the rule

There is no “Disallow AI Training on pages with ads” value. Cloudflare gives the engineering reason: the preference is expressed in robots.txt, and the set of ad-serving pages on a site is too large and too volatile to enumerate there.

That sentence is the most transferable thing in the announcement. The policy a site owner is allowed to have is bounded by the file the preference has to fit inside. Not by what the site owner wants, not by what the operator is willing to honour. By the expression mechanism. A path-list format can hold “not these paths” and cannot hold “not this category of page, recomputed hourly”. So that policy does not exist, and the reason it does not exist has nothing to do with anyone’s intent.

This is the same shape as the original problem. One crawler could not distinguish two purposes, so site owners got one decision. One file cannot distinguish a volatile page class, so site owners get no ad-scoped training rule. Both times, the mechanism set the ceiling.

The September 15 breaking change

Dated September 15, 2026, Block and “Block on pages with ads” now apply to mixed-use crawlers. Either setting now hits search as well as training. Block AI Bots is deprecated in favour of the granular controls, and Managed Robots.txt is deprecated in favour of Bot Preference Sync.

If you configured a block before that date expecting it to reach only the AI use, the behaviour changed underneath the setting. Go look.

For durable accountability, Cloudflare points at two surfaces: its Radar AI bot transparency page, which tracks what each Accountable operator actually provides, and the IETF ai-prefs working group as the destination standard.

Do this now

Open the bot settings for your primary domain and read the current value for each of the three behaviours. Not the value you remember choosing. The value that is there today, after September 15.

Then write down, in one line each, what you want for Search, Training and Agent traffic. If the three answers are not identical, you previously had no way to express them against a mixed-use crawler and you almost certainly have a stale blanket setting sitting in place of the policy you wanted. Fix that first. Then check whether your training opt-out actually reaches the operators you care about, or whether, as with Bing, it is a preference with no reader on the other end yet.

The broader discipline is one we keep arriving at from different directions: content as a licensable asset with duration as the lever, negotiating content by agent type at the request layer, agent traffic needing its own perimeter, and engineering having Cloudflare while other functions have nothing. The common thread is that a stated preference only becomes a control at a point that can see both who is asking and why. Everywhere that point is missing, your policy is whatever your mechanism happens to permit.

---

This analysis synthesizes Have it both ways: stay discoverable in search while disallowing AI training (Cloudflare, September 2026).

Victorino Group helps organizations turn declared AI policy into enforcement points that actually distinguish identity from behaviour. Let’s talk.