Does robots.txt stop AI training? What RFC 9309 says it can and cannot do

Does robots.txt stop AI training? Only if the company running the crawler chooses to honor it. Robots.txt is a 30-year-old convention, formalized as RFC 9309 in September 2022, and its own text says "These rules are not a form of access authorization." That one sentence explains most of the current fight over AI scraping, including this week's announcement of a shared AI training opt-out.

What robots.txt is

Robots.txt is a plain text file at the root of a website that lists which paths automated crawlers are asked not to visit. Martijn Koster introduced the method in 1994. RFC 9309, co-authored by Koster and three others, turned it into an IETF Standards Track document in September 2022, adding definitions, error-handling rules and caching rules without changing the core idea.

The file is organized in groups. Each group begins with one or more User-agent lines naming the crawler, followed by Allow and Disallow rules for paths. A crawler matches its own product token against the User-agent lines without regard to case. If several groups match, their rules are merged. If none match, the group for * applies, and if there is no * group, no rules apply.

The rules that trip people up

RFC 9309 sets out several mechanics that decide what a file actually does:

• Most specific rule wins. When several rules match a URL, "The most specific match found MUST be used", and specificity is measured by the number of octets in the matching path. A short Allow can be overridden by a longer Disallow.
• A size limit. Crawlers should parse at least 500 kibibytes of the file. Rules beyond that point may be ignored.
• Caching up to a day. Crawlers "SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable", so a change takes up to a day to bite.
• Missing file means open. If the file returns a client error in the 400 range, crawlers may access any resource. If the server is unreachable or returns a 500 range error, the crawler must assume complete disallow.
• Redirects. Crawlers should follow at least five consecutive redirects to find the file.

Two of those matter for site owners. A server error on your robots.txt can lock every compliant crawler out, including search engines. And a 404 does not protect anything.

It is a request, not a lock

The RFC states that the rules are a way for service owners to ask crawlers to limit which URIs they visit, and that they are not access authorization. A well-behaved crawler reads the file and follows it. A badly behaved one ignores it, and nothing in the protocol stops that. Content you need to protect belongs behind authentication, not behind a Disallow line, which can even advertise where the sensitive paths are.

For search engines, the system has worked because Google and Bing depend on a healthy web and face reputational cost for ignoring it. For AI developers, the incentives are different, and publishers have complained for years that some bots ignore the file. We cannot say from the documents we reviewed how common that is, and the figures in circulation come from vendors and interested parties.

Where AI training enters

Robots.txt controls fetching. It never said anything about what a company does with a page after fetching it. AI companies filled that gap with their own tokens. Google introduced Google-Extended in 2023 as a product token that governs whether pages Google crawled can be used for Gemini model training and grounding, separate from Search. Other companies published their own crawler names, so a site wanting out of training had to list each one.

On October 30, 2026, Google, Microsoft, OpenAI and Anthropic said they would back a shared policy token, based on work at the IETF's AIPREF working group, so that one line can state a training preference across providers. Google's post says it aims to support the control "in 2027". The draft is an individual submission that expires May 3, 2027, and it has no enforcement mechanism. Our separate news report on the announcement has the company-by-company details.

The same limits apply as before. The token expresses a preference about use. It does not block crawling, does not remove data already collected, and does not bind companies that have made no commitment. The draft's definition of training covers using content to modify a generative model's learned parameters and excludes uses such as classification and ranking.

How to use it well

If you run a site, a short checklist covers it:

1. Open yours at /robots.txt and confirm it returns a normal 200 response, not an error. 2. List the AI crawler tokens you want to treat differently from search crawlers, and write a separate group for each. 3. Keep rules specific. Remember the longest match wins. 4. Do not rely on it for anything private. 5. Revisit it when a company announces support for a new signal, and log which crawlers actually fetch the file.

If you are a reader trying to judge a claim like "we respect robots.txt", ask which user agent, from what date, and whether the company publishes crawler logs or IP ranges so site owners can verify. A promise to honor a file is only checkable if the crawler identifies itself consistently.

Our take

Robots.txt works because of norms, not code. That is both why it has lasted 30 years and why it is a weak tool against a determined scraper. The new shared token is a useful simplification for the polite majority and a test of whether the biggest AI developers will treat a request as binding. It does not turn a voluntary protocol into law. For that, publishers will keep looking to contracts, to courts and to regulation such as the EU's code of practice, which points at robots.txt as the machine-readable standard.