Policy#Web scraping#Model training
Anthropic submits the AI training opt-out token to IETF: one robots.txt line, OpenAI pledges months, Microsoft sets February 1, 2027
An Anthropic IETF draft (Oct 30) lets a site opt out of AI training via one robots.txt token; Microsoft honors it from Feb 1, 2027, OpenAI within months.
The AI training preference signal is a policy token a publisher writes into robots.txt: put it in a User-Agent group, and AI companies that crawl your site are asked to keep some or all of it out of their training data. On October 30, 2026, Kevin Kelley, Anthropic’s technical lead for standards, filed the design with the IETF as draft-kelley-aipref-training-preference-signal-00, a Standards Track submission that expires May 3, 2027. The same day, OpenAI, Google and Microsoft each posted a statement on their own blogs; Anthropic’s stance came in a mailing-list post by Kelley to the working group, where he wrote that Anthropic “has already started implementing support for this draft” (Search Engine Journal notes Anthropic’s newsroom published nothing). OpenAI, Google and Microsoft do not honor the token yet.
Key points
- The mechanism adds a token-matched group to robots.txt with no new directives and no new grammar; a wildcard
User-Agent: *group does not count, the token must be named explicitly. - The timeline: Anthropic says it has started implementing support; OpenAI says “coming months”; Google aims for 2027; Microsoft starts honoring it on February 1, 2027.
- The scope is narrower than the headlines: it covers only training that “modif[ies] the learned parameters of a generative AI model”, says nothing about crawling, and “Private agreements MAY override preferences expressed via this mechanism.”
Inside the draft
The draft is an individual submission to the IETF’s AI Preferences (aipref) working group. Writing it into a robots.txt looks no different from any other group — the User-Agent line carries the token instead of a product name. The draft’s Figure 1, copied verbatim:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
One group makes both statements at once: the whole site is out of training, except the press kit, and ordinary longest-match rules decide which line applies per path. The abstract states the design goal plainly: it “introduces no new directive names and no new grammar.” The token is the __usage_ai-training shown above — Section 4.1: “This document defines the policy token __usage_ai-training” — and the draft chose the double-underscore-prefixed form precisely because it “makes accidental matches unlikely.” Three rules are worth memorizing. Consumers must match the token exactly, apart from case. Wildcard groups “MUST NOT be interpreted as matching the policy token” — a blanket rule expresses no preference. And a robots.txt with no token group means “no preference has been expressed”, so each crawler’s proprietary tokens keep working as before, they just don’t count under this draft.
The coverage boundary is just as explicit. The aipref vocabulary defines training as using content to modify the learned parameters of a generative AI model, which leaves non-generative training outside. Crawling is not governed at all: the draft says the token “is not a directive for crawling behavior” — that belongs to the working group’s separate vocabulary and attachment documents.
What each company pledged
The statements landed within hours of each other on October 30. Cross-checked against the companies’ own posts and two independent write-ups, the timelines line up like this.
| Company | Pledge | Source, date |
|---|---|---|
| Anthropic | “has already started implementing support for this draft” | Working-group mailing list, Oct 30 |
| OpenAI | “We aim to implement the signal in the coming months” | OpenAI blog, Oct 30 |
| “We aim to support this new control … in 2027” | Search Central blog | |
| Microsoft | Honors training and grounding preferences from 2027-02-01 | Bing blog |
OpenAI adds that the signal “will not replace the controls we already provide through robots.txt” — GPTBot rules keep working. Google’s fine print is the most consequential: when a site uses the shared token without naming Google-Extended, Google will treat it “as if the publisher had disallowed Google-Extended.” That 2023-era control keeps working alongside, and Google-Extended has never affected Search inclusion. Microsoft’s post is the most concrete: a transition setting opens in Bing Webmaster Tools on December 15, 2026, honoring starts February 1, 2027, and after March 1, 2027 the legacy NoArchive and NoCache tags stop being read as preferences. Whichever way a publisher configures it, the site still appears in Bing search results.
Meta is the fifth voice, and the odd one out: communications head Andy Stone called a shared signal “an important step” but said nothing about whether Meta will honor it. Four companies gave dates; Meta gave a compliment.
A preference, not a license
The heaviest sentence in the draft is one most coverage skipped: “Private agreements MAY override preferences expressed via this mechanism.” In RFC keyword language, a licensing deal between a site and an AI company outranks the robots.txt line. Meta’s commercial terms fight earlier in October demonstrated exactly how contracts and preferences interact — the terms never mention robots.txt at all — and Benjamin Sobel (University of Wisconsin) told Nieman Lab at the time that robots.txt compliance is “completely voluntary”, like posting a “don’t look at my house” sign on your own lawn. The token changes the cost of saying no, not the obligation to care.
Chinese commentary (via Tencent’s communication-world outlet) framed the same gap as machine-readable copyright governance: its section heading says opting out is not licensing — “版权治理仍需交易层” (governance still needs a transaction layer), and a shared token makes “no” clearer without replacing permission, while pricing and scope still need contracts or collective mechanisms; forgery prevention and crawler compliance audits are questions the text does not answer. IT Home’s report stuck to the mechanism: publishers previously had to state preferences to each AI company separately, and the draft folds that into one statement. Two parallel tracks sit beside it: RSL, the licensing standard adopted by Reddit, Yahoo, Medium and others, lets sites charge crawlers; Cloudflare blocks AI crawlers by default. Three routes, three postures: express a preference, sell a license, or force the issue at the edge.
Our take
If you run a content site, the actionable part costs nothing today: write the token group into robots.txt and it takes effect automatically as companies switch on support — but per the draft’s own rules a wildcard group doesn’t count, and paths you want to reopen need explicit Allow lines. If you work in SEO or data pipelines, mind the flip side: this is a training signal, not a search signal, and both Google and Microsoft say a training opt-out does not affect search indexing or rankings. Conflating the two would cost you traffic for nothing.
Two things stay unproven. “Started implementing” comes with no verifiable scope — Kelley himself writes that rolling support out across Anthropic’s services will take time, and the draft can change materially before it expires on May 3, 2027. And a signal with no enforcement binds polite crawlers, not indifferent ones; Wikimedia’s rogues’ gallery of OpenAI agent scraping showed that the hardest traffic never waits for robots.txt. Two dates are worth circling: December 15, 2026, when Bing’s transition controls open, and February 1, 2027, when Microsoft starts honoring the token — a real test of what these pledges are worth.