


Contents (10 sections)
On 30 October, Google, Microsoft and OpenAI announced, on the same day, that they would honour one and the same line in websites' robots.txt file. That line lets you refuse to have your pages used to train their artificial intelligence. The text describing it was written by an engineer at Anthropic, the company behind Claude. Four competitors agreeing on a rule is rare. But the most important detail fits in one sentence of that text: without this line, you have refused nothing. Here is what the rule says, the catch it hides at Google, what Le Monde, Le Figaro and Leboncoin already do, and how to decide for your site.
A file from 1994 that talks to robots #
Every website has, at its address followed by /robots.txt, a small text file. Search engine robots read it before going through your pages: it tells them where they may go and where they must not enter. The convention dates back to 1994 and was only standardised in 2022. It has always relied on goodwill: the file blocks nothing, it asks.
With the arrival of generative AI, each company added its own robot: GPTBot at OpenAI, Google-Extended at Google, ClaudeBot at Anthropic, and many others. To refuse training, you had to know all these names, write them one by one and watch for new ones. Few sites did, and many did it halfway.
One line for everyone #
The text submitted to the IETF, the body that writes the internet's technical rules, proposes a shared name that each company commits to recognise. It goes where you would usually put a robot's name and works with the rules robots.txt already knows. To refuse training on the whole site, except the press area for example, three lines are enough.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press/The timeline varies by company. Microsoft will apply the line from 1 February 2027 for Bing. Google announces support during 2027. OpenAI aims for "the coming months" and keeps its current rules until then. Anthropic has started building it in. For now the text is an individual proposal, not yet a standard, and it may change before it is adopted.
Writing nothing means letting it happen #
This is the part few commentators picked up. The text states that a robots.txt file without this line expresses no preference, and that this is true of every file published to date. Microsoft puts it more bluntly: allow is the default. In other words, if you do nothing, your site has already said yes.
Another subtlety: the general rule "User-Agent: *", the one almost every site uses, does not count. Even a site that has carefully set access for all robots has, in the sense of this text, said nothing about training. To express a choice, you have to write it with the new name.
Training and answering are two different things #
This confusion is at the heart of the matter. Training means using your texts to build the model, once, without citing you. Answering means reading your page at the moment someone asks a question, and often citing it with a link. The first brings you nothing visible. The second can bring you customers.
Microsoft separates the two clearly: the shared line only concerns training, and another setting, page by page, decides whether Copilot and Bing can rely on your content to answer. At Google, generated answers in Search, AI Overviews and AI Mode, fall under yet another setting, in Search Console. Refusing training therefore does not remove you from search results. But each company draws the line in a different place, and that is where mistakes creep in.
What the big sites already do #
robots.txt files are public: just add /robots.txt to a site's address to read them. We looked at those of several major French sites on 4 November 2026. None uses the shared line yet, published five days earlier. But each has already made a choice, and those choices are very different.
Le Monde chose the list. Its file names more than 300 robots, one by one, and shuts the door on those of Google, Anthropic, Apple and Mistral. One name is missing: GPTBot, OpenAI's robot. In March 2024 the newspaper signed a multi-year agreement allowing OpenAI to use its articles. Here, robots.txt tells the story of a contract.
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: ClaudeBot
Disallow: /Le Figaro separates training from answering. GPTBot, ClaudeBot and Google-Extended are refused everywhere. But OAI-SearchBot and ChatGPT-User, the robots that read a page when someone asks ChatGPT a question, can access a selection of sections: travel, culture, fashion, deals. The newspaper accepts being cited in answers, where it decides, without feeding the models.
User-agent: OAI-SearchBot
Allow: /voyages
Allow: /culture
Allow: /style
Allow: /beaute
Allow: /joaillerie
Allow: /mode-homme
Allow: /horlogerie
Allow: /bons-plans
Allow: /elections/resultats
Disallow: /
User-agent: GPTBot
Disallow: /Leboncoin, France's leading classifieds site, takes the logic further, with comments that say it all: one block for AI search and answer robots, another for training robots. Car listings are open to the former. No listing is open to the latter.
# LLMs / AI SEARCH / RETRIEVAL bots
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
# …
Allow: /ad/voitures/
Disallow: /ad/
Allow: /
# LLMs / AI TRAINING
User-agent: GPTBot
User-agent: ClaudeBot
# …
Disallow: /ad/
Allow: /Three sites, three strategies, and one thing in common: each had to name the robots one by one, and will have to start again with every newcomer. That is exactly what the shared line is meant to simplify. Note also that Le Monde and Le Figaro block Google-Extended entirely: they therefore give up, at the same time, on answers in the Gemini app.
Should you say no? #
It all depends on what your site sells. If your texts are your product, news articles, training courses, studies or photos, refusing training protects what has value, and the shared line finally makes that refusal simple. If your site is first and foremost there to be found, like that of an SME, a firm or a tradesperson, the answer is less obvious. When someone asks an assistant which provider to choose, the AI can only cite what it knows.
In between, the line allows fine-grained choices: refuse training on your paid content and allow it on your presentation pages, or the other way round. What matters is that it is a choice, not the result of a file nobody has reread since the site was built.
In Europe, this choice carries a little more weight than elsewhere. Google points out that the line is consistent with the AI Act code of practice, which recognises robots.txt as a way to express preferences in a machine-readable form. Writing the line leaves a clear record of your decision. One limit will not change: only those who commit to it respect it.
Three template files to adapt #
Here are three starting points, to adapt to your site. The first refuses training everywhere, for a site whose content is the product.
User-Agent: __usage_ai-training
Disallow: /The second refuses training on part of the site only, for example your paid content or your resources, and allows it on the rest.
User-Agent: __usage_ai-training
Disallow: /resources/
Allow: /The third refuses training with every company except Google, to stay in Gemini's answers. At Google, the two cannot be separated: staying in Gemini also means accepting that your pages train its models.
User-Agent: __usage_ai-training
Disallow: /
User-Agent: Google-Extended
Allow: /In all three cases, keep your existing rules alongside, especially the "User-Agent: *" block: the new line adds to them, it replaces nothing.
What to check on your site #
Start by opening your site's address followed by /robots.txt in a browser. You may find old rules, set by a provider or a plugin, that already block some AI robots without anyone remembering. Check that they match what you want today.
Then decide, with whoever looks after your visibility, what you accept for training on one hand, and for assistants' answers on the other. If Google matters to you, set Google-Extended explicitly rather than letting the shared line decide for you. Finally, note the dates: the line will only take effect at Bing from February 2027, and at Google during the year. There is no rush, but nothing will happen on its own.
What this rule says about AI and the web #
For thirty years, robots.txt has told search engines where not to go. It now tells them what you accept they do with what they read. The novelty is not technical, it lies in the agreement: for the first time, the major AI companies commit to reading the same sentence. You still have to write it.
For the websites we build, the robots.txt file is part of going live, just like the sitemap: we write it with you and explain every line. The question of what AI may do with your content also comes up in any AI project, and we address it from the scoping stage of our AI consulting engagements.
Not sure what your site's robots.txt says, or what it should say? Send us its address: we will go through it with you and tell you, line by line, what it allows and what it refuses.
Sources #
- John Mueller, Google Search Central, post on the shared AI training signal (30 October 2026): support in 2027, link with Google-Extended, consistency with the AI Act code of practice.
- Krishna Madhavan, Microsoft Bing, Your content, your choice: clearer preferences for training and grounding (30 October 2026): allow by default, page-level answer setting, timeline.
- Kevin Kelley (Anthropic), Machine-Readable Training Preference Signal, IETF draft (30 October 2026): syntax, no preference without the line, "*" rule excluded.
- Matt G. Southern, Search Engine Journal, article on the shared AI training opt-out (1 November 2026): OpenAI's and Anthropic's commitments, setting for AI answers in Google Search.
- José Billon, Blog du Modérateur, article on the shared robots.txt rule against AI training (2 November 2026), in French: each company's commitments.
- IETF, RFC 9309, Robots Exclusion Protocol (September 2022): the 1994 convention, standardised.
- The robots.txt files of lemonde.fr, lefigaro.fr and leboncoin.fr, retrieved on 4 November 2026.
- Siècle Digital, article on the multi-year agreement between OpenAI and Le Monde (14 March 2024), in French.






