On October 30, 2026, Google said it will start honoring a shared signal in robots.txt that lets a site owner ask for their content to stay out of AI model training. OpenAI published a similar announcement the same day, and Microsoft and Anthropic said they would adopt the proposal too. The signal comes from a draft by a working group at the IETF, the internet's standards body, that deals with how sites state their preferences about AI.
robots.txt is a short text file at the root of a website that tells crawlers from search engines and AI companies which pages they may visit. Anyone can read it in a browser by adding /robots.txt to the site's address.
One line instead of a list of crawlers
Until now, a business that wanted to keep its content out of training had to know the name of each company's crawler and write a separate rule for every one of them. OpenAI said the shared signal removes that chore, since one preference reaches every participating provider at once. According to Google's documentation, the line will start with user-agent: __usage_ai-training, followed by an ordinary disallow rule.
Google's documentation puts support in early 2027. OpenAI said it will implement the signal in the coming months and share details when it does. The proposal is still an IETF draft, so the wording may change before it is approved. Each company's existing rules will keep working alongside the shared signal.
The cost at Google
This is the detail a business owner needs to know. Google said that a site opting out of training through the shared signal, without a specific rule for its Google-Extended crawler, will be treated as if it had disallowed Google-Extended. That setting covers more than training. Google's documentation says Google-Extended also decides whether a site's content can be used for grounding in the Gemini apps, meaning whether Gemini can draw on the site as a source when someone asks it a question. Google wrote that sites which allow Google-Extended are eligible to appear in those responses and receive clicks from them.
At Google the two cannot be split: the same setting governs training and appearing as a source in Gemini. A hair salon, a renovation contractor or an online shop that blocks training cannot be the site Gemini points to when someone asks it about that kind of service.
Microsoft took a different route. It announced new controls for Bing that let a site choose separately whether its content is used for training, for grounding AI answers, or for both, while staying eligible for Bing search results.
Search rankings stay put
The first worry for most owners is Google search results. Google's documentation states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. A site that blocks training will keep appearing and ranking in Search as before.
Before adding the line
There is nothing to change until support begins, because Google and OpenAI do not read the signal yet. It is worth opening the file now to see which rules are already there. A site that already blocks Google-Extended, perhaps because whoever built it added the rule without thinking about Gemini, is already left out as a source in Gemini's answers today.
The choice depends on what the site sells. A business whose content is the product, such as paid guides, courses or recipes, can gain from blocking. A business whose site exists so customers can find it loses a source of visitors when it disappears from AI answers.