Blocking GPTBot Does Nothing to ChatGPT Search. Check the Other Token.

There's one mistake that removes a site from AI answers more reliably than any content problem, and it's usually three lines in a file nobody has opened since 2023.

robots.txt now governs at least four distinct kinds of automated visitor, and they get treated as one. Getting the categories wrong produces both failure modes: sites that think they're protected and aren't, and sites that think they're visible and aren't.

Four categories, one file

AI search indexers build the indexes assistants cite from. Block these and you are removed from AI search answers. Roughly ten tokens matter here: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, MistralAI-Index, DuckAssistBot, YouBot, Amzn-SearchBot, Google-CloudVertexBot, xAI-Bot.

AI training crawlers collect content to train foundation models: GPTBot, ClaudeBot, GoogleOther, CCBot, Amazonbot, meta-externalagent, Bytespider, AI2Bot and others. Blocking these is a licensing and positioning decision. It does not affect whether AI search engines cite you today.

User-triggered fetchers load a page live because a human asked an assistant about it: ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User, MistralAI-User. Several operators state explicitly that robots.txt may not apply here, on the reasoning that a person requested the page.

Control tokens aren't crawlers at all. Google-Extended and Applebot-Extended never appear in server logs. They're robots.txt-only switches governing how data an existing crawler already collected may be used downstream.

The consequential collapse is between the first two categories, because the naming actively misleads. GPTBot sounds like the ChatGPT bot. It's the training crawler, and blocking it has zero effect on OAI-SearchBot: separate user agent, separate crawling infrastructure, separate robots.txt directives. The same split exists at Anthropic, where ClaudeBot trains and Claude-SearchBot indexes for citation.

So the 2023-era "block everything AI" line, still sitting in a great many robots.txt files, frequently means: keep training out, and also keep the citation index out.

How widespread is that? Nobody has a number worth putting in a deck. Published estimates for "sites blocking at least one major AI crawler" run from roughly 5% to roughly 45%, and the gap is not disagreement about the data, it is disagreement about the sample: the alarming figures come from the top thousand news and publisher domains, which had a licensing fight your site did not. Cloudflare Radar publishes the live directive mix across its network if you want the shape of it. What is not in dispute is which file decides the question for your domain, and that one you can read in a minute.

The Google-Extended myth

Disallow: Google-Extended is widely believed to remove a site from AI Overviews. It doesn't.

Google-Extended controls whether Googlebot-crawled data may be used for Gemini training and grounding. AI Overviews and AI Mode ride the normal Google Search index. The only way out of them is out of Search altogether, a trade almost nobody wants.

Applebot-Extended works the same way. Blocking it opts you out of Apple Intelligence training while keeping you in Siri, Spotlight and Safari suggestions, because those run on Applebot proper.

Control tokens govern use, not access. Read them as access rules and you end up with the downside of a block and none of the intended effect.

Check what your file actually says

Reasoning about robots.txt from memory is how these errors persist. The AI crawler access checker fetches your live file and evaluates each of 35 known AI user agents against it, grouped by category, with the consequence of each verdict spelled out.

Note what sits under each verdict: line 2: Allow: / (via User-agent: *). Not a green tick, but the specific rule that decided it, evaluated with the same precedence real crawlers use (most-specific group wins, longest path match, allow beats disallow on ties, per RFC 9309). A verdict you can't trace to a line is a verdict you can't fix.

Two further details in how it reports, both deliberate:

It says "requested not to crawl" rather than "blocked" for agents whose compliance is unverified or documented as non-compliant. Robots.txt is a request, not a fence. Three of the tokens in the roster are marked as not respecting it (Bytespider, DeepSeekBot and Perplexity-User), and for those, only WAF or bot-management rules do anything. Calling that "blocked" would be a lie with security consequences.

It flags deprecated tokens separately. anthropic-ai, claude-web and FacebookBot no longer appear in their operators' documentation. Keeping them in a block list is harmless; believing they're doing something isn't.

If you need to write the file rather than audit it, the AI robots.txt generator produces one from explicit per-category choices, so allowing search indexers while blocking training crawlers becomes a checkbox instead of a research project. Both tools sit in the free tools hub with twelve others.

A default that fits most brands

For a business that wants to be cited by AI assistants and isn't selling a content-licensing position:

Category	Default	Why
AI search indexers	Allow all	This is the citation index. Blocking is self-removal.
User-triggered fetchers	Allow	Someone is actively asking an assistant about your page.
Training crawlers	Your call	No effect on citation. Decide on licensing grounds.
Control tokens	Your call	Governs downstream use only, not access or visibility.

The training-crawler decision deserves a genuine argument instead of a default, and the economics have moved. Crawl-to-referral ratios are lopsided: Anthropic's ClaudeBot has been reported crawling on the order of twenty thousand pages for every referral sent back to publishers. If you're a publisher whose business is pageviews, "they take a lot and send almost nothing" is a real position, and blocking training crawlers costs you no citations.

If you're a brand whose business is being chosen, the calculation inverts. You aren't selling pageviews; you want to be in the answer. Blocking training crawlers there means paying a cost for a principle you don't hold.

Auditing this properly, in order

1. Fetch your live robots.txt, not the one in your repo. Redirects, CDN rules and platform defaults all interfere. Read the bytes the crawler reads.
2. Separate search indexers from training crawlers before you interpret a single line. Any conclusion drawn before this step is unreliable.
3. Check your CDN and WAF too. Cloudflare, Fastly and Vercel all offer bot-blocking that operates above robots.txt. A file that allows OAI-SearchBot while your edge returns 403 to it is a block that never appears in the file.
4. Look at server logs for the tokens you allow. Access permitted and access used are different facts. If PerplexityBot never appears in your logs, permission isn't your problem.
5. Then measure citations per engine. Crawler access is a mechanism, and the outcome is what you're after. In our demo scan, the brand's own domain accounted for 3.2% of 63 cited URLs and every one of those citations came from a single engine, a per-engine pattern only a per-engine citation report reveals.

Step 5 is where crawler work either pays off or reveals itself as the wrong diagnosis. Unblocking an indexer can raise citation share without moving your mention rate at all, because the model already knew you existed and simply couldn't fetch you. The reverse pattern (good crawler access, low citation share) means the retrieval layer can reach you and is choosing someone else's page, which is a content and coverage problem no robots.txt edit will fix.

FAQ

Does blocking GPTBot remove me from ChatGPT search results? No. GPTBot is OpenAI's training crawler. ChatGPT search visibility depends on OAI-SearchBot, a separate user agent with separate infrastructure and separate robots.txt rules. Blocking one has no effect on the other.

Does blocking Google-Extended remove me from AI Overviews? No. Google-Extended is a control token governing whether Googlebot-crawled data may be used for Gemini training and grounding. AI Overviews and AI Mode ride the normal Google Search index, so the only way out of them is out of Search itself.

Which AI crawlers should a brand that wants visibility allow? All the search indexers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, MistralAI-Index, DuckAssistBot and their peers. These build the indexes assistants cite from. Whether to allow training crawlers such as GPTBot, ClaudeBot and CCBot is a separate licensing decision with no effect on citation.

Does robots.txt actually stop AI crawlers? Only the ones that choose to comply. Several documented crawlers ignore it, and some operators state that user-initiated fetches may bypass robots.txt because a human asked for the page. Non-compliant traffic needs WAF or bot-management rules; a text file won't touch it.