Google backs a shared robots.txt signal to opt out of AI training everywhere, and it won't land until 2027

Google said this week that it plans to support a single, shared robots.txt signal for opting out of AI training, built through an IETF working group so that one instruction could eventually cover every participating AI company instead of just Google's own models.

The announcement, posted October 30 on the Google Search Central Blog by Search Advocate John Mueller, describes a new policy token that website owners could add to robots.txt to set a broad AI-training preference across providers in one move, instead of maintaining a separate disallow line for every crawler. The work draws on the AIPREF working group at the Internet Engineering Task Force, the same standards body that governs robots.txt itself.

What Google actually committed to

Google has offered a training opt-out since September 2023, when it introduced Google-Extended, a robots.txt token with no crawler of its own that lets a site block its content from training future Gemini models and from grounding answers in Gemini Apps, without affecting Search indexing or ranking. That control still works the same way: one line, one company. What changes is that Google now says it will also honor a shared token. If a site uses that token to opt out of training and does not separately address Google-Extended, Google will treat the broad opt-out as a Google-Extended disallow too.

"Support for this proposal across major AI providers will enable a future standard that is cross-industry, robust, and durable."

That is the aspiration, not the current state. The mechanism comes out of an IETF Internet-Draft, co-written by Google's Gary Illyes and Mozilla's Martin Thomson, that would add what the draft calls a Content-Usage rule to the robots.txt standard, Search Engine Journal reported on October 22. Illyes has described robots.txt itself as "an access preference protocol" that governs crawl access, not what happens to data afterward -- exactly the gap the new rule is meant to close. As of that report, the draft was still a working-group document, not yet sent to the IETF's Steering Group for final review. Google's own post is explicit that it aims to support the shared control "along with other AI providers" only in 2027.

Opting out of training is not the same as opting out of AI answers

One line in the post deserves a software marketer's attention. Sites that specifically allow Google-Extended, Google says, stay eligible for responses and clicks through grounding in Gemini Apps. Read plainly, that means using the new blanket token to block training, without separately re-allowing Google-Extended, could also cost a site visibility and referral clicks inside Gemini's grounded answers -- a channel software marketers have spent the past two years trying to get cited in, not shut out of. Existing per-vendor controls keep functioning alongside the new signal, so nothing in today's robots.txt file breaks on its own. But a one-click opt-out applied broadly by a legal or security team, without input from whoever owns AI visibility, could quietly trade away the citations that team is chasing.

Compliance with any robots.txt directive, old or new, is voluntary. A crawler that ignores it faces no technical enforcement, only reputational and regulatory pressure -- the kind Google's post leans on when it ties the proposal to the EU's Code of Practice for General-Purpose AI Model Providers, which Google says already points to robots.txt as the primary channel for publishers to express AI-training preferences under the EU AI Act.

The shared token also only covers the narrow question the IETF's AIPREF group has reached consensus on so far: whether content can be used to train a model. A separate, still-unsettled piece of that same working group's vocabulary draft would eventually let sites signal preferences about other AI uses, such as search grounding or retrieval, but Google's post says only that multistakeholder talks on the remaining vocabulary continue. For now, training is the only lever this new token pulls.

Who this affects, and what to do now

It affects anyone who manages a software company's public content -- docs, blogs, comparison pages, changelogs -- and has an opinion on whether that content should train someone else's model. It also affects anyone tracking AI citations and referral traffic as a channel, since the fine print links training permissions to grounding eligibility. Neither group typically owns robots.txt directly; that file usually belongs to IT, DevOps or a webmaster, which is exactly how a training-access decision ends up made without a marketing voice in the room.

• Find out who edits your robots.txt today and get looped into any future change tied to AI training -- the decision trades training exposure against AI-answer visibility, which is a marketing call as much as a technical one.
• Audit your current per-vendor AI tokens now. Nothing about the shared signal changes before 2027 at the earliest, so Google-Extended and any equivalent lines for other AI crawlers still do the actual work.
• If your organization adopts the shared opt-out once it ships, decide explicitly whether to also keep Google-Extended allowed -- Google ties that choice directly to eligibility for citations and clicks from Gemini grounding.