One robots.txt Line to Say No to AI Training: What OpenAI, Google, Microsoft and Anthropic Just Agreed

Yesterday we wrote about Meta's new business terms and its website crawlers, and how every AI company asks you to block its crawler by a different name. On October 30, 2026, OpenAI, Google and Microsoft announced they will support one shared signal instead, and Anthropic, whose standards lead wrote the proposal, says it has already started building support. If your business has a website, a blog or an online catalogue, this is the simplest way yet to tell AI companies whether they may train their models on your pages. Here is what it is, what it is not, and what to do. Every fact is linked.

1. What was announced on October 30

The signal is a new "policy token" for robots.txt, the plain text file at the root of a website that tells automated visitors what they may do. It comes from an Internet-Draft, draft-kelley-aipref-training-preference-signal, filed at the Internet Engineering Task Force (IETF), the body that writes many of the internet's technical standards. The Verge reports that its author, Kevin Kelley, is a technical lead for standards at Anthropic.

• Google said it plans to support "a shared AI training control in robots.txt" so websites can "express their AI training preferences across multiple AI providers in a single step" (Google Search Central Blog).
• OpenAI said it "will honor a common signal in robots.txt" about the use of content for "foundational generative AI training" (OpenAI).
• Microsoft said it is "working with other industry participants to bring a shared AI training preference to site owners and publishers" (Microsoft Bing Blog).
• Anthropic: Kelley wrote on the IETF mailing list that "Anthropic has already started implementing support for this draft", adding that it will take time to cover all its services (IETF ai-control list).

Until now, a site that wanted to opt out of AI training had to list each company's crawler or control by name and keep that list up to date. As Search Engine Roundtable puts it, the new control lets publishers state their preference "in a single step in robots.txt, instead of listing each one individually."

2. How the shared line works

The token is __usage_ai-training. It goes in the place where a crawler's name normally goes. This example comes straight from the draft:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

According to the draft, that means "no AI training" for the whole site, except the /press-kit/ folder, where training is welcome. Three details matter:

• It is about use, not access. The draft says plainly: "This policy token is not a directive for crawling behavior." It records your preference about training. If you want to stop a crawler from visiting at all, you still need rules for that crawler's own name.
• "User-Agent: *" does not count. Under the draft, a general rule for all bots is never read as a training preference. You must name the token.
• Company-specific rules still win. The draft says specific expressions "SHOULD take precedence over general ones", and that private agreements may override the signal.

"AI training" has a narrow meaning here, taken from the IETF working group: "Using an asset to modify the learned parameters of a generative AI model." Training models that only classify, rank or score content is not included.

3. Who supports it, and from when

This is a promise, not a switch that flipped on October 30. The timelines differ:

Company	What it said	When
Anthropic	Has "already started implementing support"; full coverage of its services will take time	Under way
OpenAI	Will honour the signal; its existing crawler controls "will continue to apply"	"In the coming months"
Google	An opt-out with the token counts as disallowing Google-Extended, unless Google-Extended is named separately	Aims for 2027
Microsoft	Will honour it as Bing crawls, plus new page-level settings	From February 1, 2027
Meta	Spokesperson Andy Stone signalled support but did not say Meta will honour it (The Verge)	Not stated

Sources: Kelley on the IETF list, OpenAI, Google, Microsoft, The Verge. Stone told The Verge that "creating a common signal that conveys such preferences to multiple providers at once is an important step."

4. Training is not the same as being cited

For a business, this is the part worth slowing down for. Saying no to training is different from saying no to being used in AI answers, which Microsoft calls "grounding": pulling your page into a chatbot or AI search answer, usually with a link back to you.

• Google says existing company controls keep working alongside the shared signal, and that sites which specifically allow Google-Extended "are eligible to appear in responses and receive clicks via grounding in Gemini Apps" (Google).
• Microsoft is adding a separate, page-level AI Grounding setting with three choices: Allow (the default), DisallowExtended (only Microsoft's own products such as Copilot, Microsoft 365 and Bing's AI answers) and Disallow. It warns that Disallow or DisallowExtended "will significantly reduce your content's visibility in AI-generated experiences", while normal Bing search results are not affected (Microsoft Bing Blog).
• Microsoft is also clear that "neither choice guarantees inclusion, traffic, or payment."
• One thing to watch: because Google will read the shared opt-out as a Google-Extended disallow, a business that wants to stay eligible for Gemini citations should check Google's documentation when its support goes live in 2027. Google's post does not spell this case out.

So a shop that wants customers to find it through AI assistants may choose "no training, yes to citations". A course creator or publisher whose content is the product may choose to say no to both. There is no single right answer.

5. What Indian website owners can do now

For D2C brands, coaching institutes, travel agents, manufacturers with product catalogues and creators with blogs, robots.txt is usually a two-minute edit. Most website builders and SEO plugins have a robots.txt editor; on custom hosting it is a plain text file at the root of your domain.

1. Decide training and citations separately. Write down your choice for each before you touch the file.
2. Add the shared line if you want to opt out of training. To opt out site-wide, the two lines are User-Agent: __usage_ai-training and Disallow: /. Microsoft's own example uses the same lines (Microsoft).
3. Keep your existing company-specific rules. The shared token is not live everywhere yet, so the named rules you already use (for example, the Meta crawler rules from our last guide) are still what works today. OpenAI and Google both say their existing controls continue.
4. If you used Bing's old NoArchive or NoCache tags, note the dates. Microsoft says those tags stop counting as a training or grounding preference after March 1, 2027; temporary grounding settings open in Bing Webmaster Tools on December 15, 2026. For one page, Microsoft's training opt-out tag is <meta name="Bingbot" content="ai-training:disallow"> (Microsoft).
5. Remember it is a request, not a lock. Like all of robots.txt, the signal depends on companies choosing to respect it. Anything confidential, such as dealer price lists or customer files, belongs behind a login.

The bottom line

For the first time, the companies behind ChatGPT, Gemini, Copilot and Claude have agreed on one way to read "don't train on my site". It is a draft, not yet a final standard, and only Anthropic says it has started; OpenAI says months, Microsoft February 2027, Google 2027. Adding the line now costs nothing and states your choice early. Just don't remove the crawler-specific rules you already have, and think about citations separately from training. For the Meta side of this story, read Meta's new business terms and your website.

FAQ

What is the __usage_ai-training line in robots.txt?

It is a shared "policy token" proposed at the IETF. A robots.txt group that starts with "User-Agent: __usage_ai-training" followed by "Disallow: /" tells participating AI companies you prefer that your content is not used to train generative AI models.

Does it work today?

Partly. Anthropic says it has started implementing support. OpenAI says it will honour the signal in the coming months, Microsoft from February 1, 2027, and Google aims for 2027. Until then, keep your existing company-specific robots.txt rules.

Will it stop AI crawlers from visiting my website?

No. The draft says the token is not a directive for crawling behaviour; it only states your preference about AI training. To block a crawler from visiting, you still need rules for that crawler by name.

Will opting out of AI training remove my site from Google or Bing search?

Google and Microsoft say these AI controls do not affect eligibility for normal search results. Microsoft has a separate grounding setting that controls whether pages can inform AI answers, and says disallowing it will reduce visibility in AI experiences.

Does Meta honour the shared signal?

Not yet confirmed. The Verge reports that Meta spokesperson Andy Stone signalled support for a common signal but did not say whether Meta will honour it.

Sources

1. Google Search Central Blog (John Mueller): Supporting a shared AI training control for websites (Oct 30, 2026)
2. OpenAI: Making it easier for websites to express AI training preferences (Oct 30, 2026)
3. Microsoft Bing Blog (Krishna Madhavan): Your content, your choice: clearer preferences for training and grounding (Oct 30, 2026)
4. IETF Internet-Draft (Kevin Kelley): draft-kelley-aipref-training-preference-signal-00 (Oct 2026)
5. IETF ai-control mailing list: Kevin Kelley: New draft: draft-kelley-aipref-training-preference-signal-00 (Oct 30, 2026)
6. The Verge (Emma Roth): The biggest names in AI are backing a new way to opt out of scraping (Oct 30, 2026)
7. Search Engine Roundtable (Barry Schwartz): Google Adopts New AI Training Control (Oct 30, 2026)

Quotes from Google, OpenAI, Microsoft and the IETF draft were checked on their own pages on October 31, 2026. The draft is a work in progress and may change. Statements by Meta's spokesperson are as reported by The Verge. This article is general information, not legal advice. The cover image is our own illustration.