One line against AI training: the big providers agree on a shared signal. What it means for your website

TL;DR. On October 30, OpenAI, Google and Microsoft announced that they will honor one shared signal in robots.txt. Two lines will then be enough to tell every participating provider: please do not use this content for training. The draft comes from an Anthropic employee and sits with the internet standards body IETF. Fixed dates exist only for 2027, and the signal stays voluntary. On October 31 we read the robots.txt files of the 40 DAX companies: none of the 33 readable files says no to training, and four explicitly invite AI crawlers. For mid-sized companies the signal is above all a decision aid. It separates "train on my content" from "show me in AI answers". Decide that for each part of your website, not across the board.

What was announced on October 30

robots.txt is a small text file in which a website states which programs may fetch which pages. Until now, anyone who wanted to rule out AI training had to name each provider separately: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google and more. Microsoft had no name of its own for this; Bing read the page-level settings NoArchive and NoCache.

The new draft introduces one shared name: __usage_ai-training. It looks like a crawler, but it stands for a purpose. This is what an entry looks like that refuses training on the whole website and only opens up the press section:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press/

Under the definition of the IETF working group, this covers only the training of generative models. Models that merely sort or score are not included. The commitments in detail:

• OpenAI says in its announcement that it aims to honor the signal "in the coming months". Rules for GPTBot stay in force.
• Google plans to support it in 2027, according to Search Central. Google points to the EU Code of Practice for providers of large AI models, which recognizes robots.txt as a machine-readable method.
• Microsoft names February 1, 2027 in the Bing blog. It adds a second setting per page: whether the page may serve as a source for answers in Bing, Copilot and Microsoft 365.
• Anthropic provides the author of the draft; we found no date of its own. According to SiliconANGLE, Meta signaled support but made no commitment.

Why this is more than a formality

For German companies the question has a legal core. Under Section 44b of the German Copyright Act, freely accessible content may be copied for text and data mining unless the rights holder has reserved this. For content online, that reservation only counts if it is machine-readable. A signal that all major providers read the same way makes the reservation clearer than a sentence in the legal notice. Whether it is enough in a dispute is not something the signal answers. That remains a question for lawyers.

More important day to day: training and visibility can be separated. The signal only concerns training. Whether ChatGPT, Gemini or Copilot fetch your pages to answer a question and point to you is governed by other entries.

The catch

• Voluntary. robots.txt is a request. Anyone who ignores it is not technically stopped.
• Not retroactive. What has already been collected and trained on stays in. Anthropic also writes about its crawler that a block applies to future content.
• Not a standard yet. The draft is an individual proposal that the IETF explicitly does not endorse. It expires on May 3, 2027 unless someone renews it.
• The asterisk does not count. A rule for User-Agent: * expresses no training preference, according to the draft. If you write nothing, you have said nothing.
• The Google trap. If you set the signal without naming Google-Extended, you are treated as if you had blocked Google-Extended. But Google also uses Google-Extended to decide whether pages appear as a source in answers in the Gemini app. If you want to stay visible there, allow Google-Extended explicitly.
• The Bing deadline. From March 1, 2027, NoArchive and NoCache no longer count as a training opt-out at Microsoft.

We checked: what DAX companies put in their files today

On October 31 we fetched the robots.txt from the corporate websites of all 40 DAX companies, once per website and read-only. The index composition comes from Wikipedia. 33 files were readable, 7 sat behind bot protection or did not respond.

• None of the 33 blocks a major training crawler, neither GPTBot nor ClaudeBot, Google-Extended or CCBot.
• None uses the new signal. One day after the announcement, that is no surprise.
• Four explicitly invite AI crawlers. Three with their own allow rules for GPTBot, ClaudeBot or Google-Extended. One with the entry Content-Signal: ai-train=yes introduced by Cloudflare.
• One blocks only ByteDance's crawler, as part of a list of unwanted programs. 28 do not mention AI at all.

Publishers look different. Ahrefs analyzed 24,550 websites from Germany, Austria and Switzerland in August. 2.27% block GPTBot. Among 28 large news and publishing sites, it is 61%.

So the line does not run between large and small. It runs between "my text is advertising" and "my text is my product".

What this means for mid-sized companies

For most of their website, most mid-sized companies are on the side of the corporations. Product pages, references and job ads should be found, including by AI assistants that customers now ask instead of a search engine. But there are often parts that are closer to a product: expert articles, training material, detailed manuals, your own photos and databases. For those parts, a no to training can make sense without the whole website disappearing from AI answers. That per-path separation is exactly what the new signal allows.

What you should do now

1. Look at your robots.txt. Add /robots.txt to your domain and open the address in a browser. Check whether AI crawlers are named and whether that still matches your intent.
2. Decide per section. Split your website into "should be found" and "is our product". Only the second part is worth a no to training.
3. Set the signal and keep the old entries. Programs that do not know the signal skip it. For now, only the per-provider rules work reliably, for example for GPTBot, ClaudeBot and Google-Extended.
4. Allow visibility on purpose. If you want to be linked in Gemini answers, allow Google-Extended explicitly. With Microsoft you control this per page from February.
5. Put a date in February 2027 in your calendar. That is when Microsoft's new rule applies, and in March NoArchive and NoCache lose their effect. A good moment to check the file again.

For the first time, the big providers speak the same language on this question. That does not make the decision for you, but it makes it simpler. If you want to know what your website allows today and how it shows up in AI answers, we are happy to look at it with you.