One robots.txt Line for AI Training Opt-Out: What It Does and Doesn't Block

As of October 30, 2026, there is one robots.txt line that tells Google, Microsoft, OpenAI and Anthropic not to train AI models on your content: a User-Agent: __usage_ai-training group with a Disallow rule. It controls training only. It does not remove you from search, AI Overviews or AI answers, and that separation is the most useful thing to come out of this week for brands that want to stay visible in AI search.

Until now, most sites that wanted out of AI training had to block entire crawlers, and many blocked the wrong ones. The new signal lets you say "do not train on this" without also saying "do not cite me".

What was announced

On October 30, Google, Microsoft, OpenAI and Anthropic each said they would support a shared training preference signal described in an IETF draft, draft-kelley-aipref-training-preference-signal, authored by Kevin Kelley of Anthropic. It is an individual submission, not yet a working group document, so the details can still change.

The syntax uses plain robots.txt grammar, with no new directives. The example in the draft, as reported by fireup.pro, looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That block says: do not use the site for AI training, except the press kit. Normal path matching applies, so the longest matching rule wins.

Two details matter more than they look:

• A wildcard group does not count. User-Agent: * is not read as the training token. If your file has no __usage_ai-training group, you have expressed no training preference at all.
• It is not a crawl rule. The draft says the token governs how content is used, not whether it is fetched. Crawler-specific rules still control access.

Who committed to what, and when

• Google will treat a training opt-out via the token as if Google-Extended were disallowed, as long as you have not named Google-Extended explicitly. An explicit Google-Extended group still wins. Google says support comes in 2027, per Search Engine Roundtable.
• Microsoft is going further. Bing now splits AI Training from AI Grounding (using your content to generate AI answers) and says both preferences will be honored from February 1, 2027, while your Bing search eligibility stays the same (Search Engine Roundtable).
• OpenAI says it will implement the signal in the coming months, alongside its existing robots.txt controls.
• Anthropic says it has already started implementing support, with rollout across services taking time.

Participation is voluntary and unenforced, and other large AI companies are not on the list. Treat the token as a strong, widely read preference, not a lock.

What it does not block

This is where teams are likely to get it wrong, so it is worth being precise.

• Search indexing and rankings. Unaffected at Google and Bing.
• Google AI Overviews and AI Mode. These run on the regular Search index. Google handles them with a separate Search Console toggle that it says was rolled out to all sites worldwide as of August 31, 2026. The training token does not touch it.
• Bing grounding. Microsoft keeps grounding as a page-level preference, set with a meta tag or X-Robots-Tag header rather than robots.txt. Sources report values like ai-grounding:disallow and ai-grounding:disallowextended, the latter limiting grounding to Microsoft's own products (how2shout). Check Bing's documentation for exact syntax before shipping.
• Content already collected. The draft applies to content as it is retrieved. It says nothing about data gathered before you changed the file.

The one overlap to watch: because Google maps the token to Google-Extended, and Google-Extended also covers grounding in Gemini Apps, a blanket training opt-out may reduce your presence in Gemini answers. If Gemini visibility matters to you, add an explicit Google-Extended group that says what you actually want.

Why this matters for AI visibility

We have argued before that most AI crawler blocks were never a decision. They came from a CDN default, a security plugin, or a copy-pasted list that lumped training bots, search bots and live-fetch agents together. A brand that only wanted to keep its content out of model training often ended up invisible in ChatGPT search and Perplexity answers too.

The shared token gives that brand a cleaner option. You can opt out of training with one group and leave OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and Bingbot alone. If you followed our robots.txt and llms.txt guide and blocked GPTBot or ClaudeBot to stop training, those rules still work. The token adds a provider-neutral way to say the same thing.

There is a deadline hiding in Bing's timeline too. Sites using noarchive or nocache to keep content out of Bing's AI features will see those directives stop counting as training or grounding preferences on March 1, 2027. According to fireup.pro and ppc.land, temporary grounding settings open in Bing Webmaster Tools on December 15, 2026, for sites already using those directives, and expire by September 30, 2027.

What to do this month

1. Audit your current robots.txt. Run the Robots.txt Tester against your pricing page and a top blog post. Note which AI crawlers are blocked and whether each block was intentional.
2. Decide training and visibility separately. Write down two answers: may AI companies train on our content, and do we want to be cited in AI answers? For a brand that depends on being found, the second answer is usually yes, whatever the first one is.
3. If you opt out of training, add the token instead of widening blocks. Add a __usage_ai-training group, then remove any search or live-fetch crawler blocks that existed only to stop training.
4. Set Google-Extended on purpose. If you want to stay in Gemini Apps answers, name Google-Extended explicitly so the token's default mapping does not decide for you.
5. Check for noarchive and nocache. If you use them for AI reasons, plan the move to Bing's page-level grounding preference before March 1, 2027.
6. Verify the result. After any change, confirm crawlers can still reach your key pages with the Crawler Access checker, then watch whether your brand keeps showing up in answers.

That last step is the one most teams skip. A robots.txt edit is invisible until a buyer asks ChatGPT, Claude or Gemini about your category and you are missing. GetBotRank's Visibility Tracker runs those buyer questions on a schedule, so a bad crawler rule shows up as a falling mention rate instead of a mystery. If you want a quick baseline first, the free AI visibility audit checks crawler access and AI readiness on any page, no signup needed.

The bigger picture: AI companies are finally treating "train on it" and "cite it" as separate permissions. Brands that make both choices on purpose, rather than inheriting them from a default blocklist, will be easier to find in AI search next year.