AI Crawler robots.txt Generator
Pick a policy, fine-tune per bot, and get a commented robots.txt for 25+ AI crawlers. Runs entirely in your browser — nothing is uploaded.
Toggle a bot on to block it (Disallow: /). Anything left off is allowed — in robots.txt, absence of rules means access.
AI search indexers
Build the indexes AI assistants cite from. Blocking these removes you from AI search answers — most brands that want AI visibility should allow them.
Indexes for ChatGPT search. OpenAI says access is required to appear in ChatGPT search results.
Indexes for Claude's search features. Blocking removes you from Claude search citations.
Perplexity's search indexer (declared not used for foundation-model training).
Crawls on behalf of Vertex AI Search customers building site-specific agents.
Powers Siri, Spotlight, and Safari suggestions. Blocking removes you from those surfaces.
Indexes for Amazon Rufus/Alexa search answers.
Registry-observed xAI token; undocumented by the operator.
Mistral's search indexer.
Powers DuckAssist AI answers. Opt-out takes ~72h and doesn't affect organic rankings.
You.com search index and LLM retrieval.
User-triggered fetchers
Fetch a page live because a human asked an assistant about it. Several operators say robots.txt may not apply to these, since a user initiated the request.
Live fetches when a ChatGPT user asks about your site. OpenAI notes robots.txt rules may not apply to user-initiated fetches.
Fetches pages a Claude user asks about. Anthropic states it respects robots.txt.
Perplexity's docs state this user-triggered fetcher generally ignores robots.txt.
Fetches links for Meta AI assistants; Meta says it may bypass robots.txt for user-initiated requests.
Live fetches for Alexa user requests.
Fetches pages for Le Chat users; explicitly not used for training.
AI training crawlers
Collect content to train foundation models. Blocking them is a licensing/positioning choice — it does not affect whether AI search engines cite you today.
Opts your content out of OpenAI model training. Does not affect ChatGPT search visibility.
Anthropic's training crawler. Also honors the non-standard Crawl-delay directive.
Google's generic crawler for internal research and one-off crawls.
Meta's AI training and product-improvement crawler.
Trains ByteDance LLMs (Doubao). Widely documented ignoring robots.txt — a real block needs WAF/bot management.
Builds the open Common Crawl dataset — the largest upstream source of LLM training corpora.
Feeds Alexa answers and Amazon model training. Ignores Crawl-delay.
Cohere training crawler; robots.txt compliance is disputed between registries.
xAI publishes no crawler docs; Grok retrieval traffic is reported with spoofed browser UAs.
Crawls for the open Dolma training dataset.
Marked non-compliant by crawler registries — robots.txt is advisory only here.
Knowledge-graph extraction resold via API; fetches on behalf of customers on demand.
Opts out of Webz.io's crawl-data marketplace used for LLM training (successor to omgili).
Decentralized index used for model training.
Control tokens (not crawlers)
Robots.txt-only switches that govern how data from an existing crawler may be used. They never appear in server logs.
Governs use of Googlebot-crawled data for Gemini training/grounding. Blocking it does NOT remove you from AI Overviews or AI Mode — those ride the normal Search index.
Governs use of Applebot-crawled data for Apple Intelligence training. Blocking it keeps you in Siri/Spotlight.
Tokens no longer in operator docs. Off by default — only worth adding for belt-and-suspenders block lists.
# AI crawler rules
# Generated with https://whereamimentioned.com/tools/ai-robots-txt-generator
# AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: GoogleOther
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: cohere-training-data-crawler
Disallow: /
User-agent: GrokBot
Disallow: /
User-agent: AI2Bot
Disallow: /
User-agent: DeepSeekBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: Webzio-Extended
Disallow: /
User-agent: Timpibot
Disallow: /
# Control tokens (not crawlers)
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Note: Bytespider, DeepSeekBot are reported to ignore robots.txt.
# These rules are a formal signal only — enforce with WAF/bot management.
Currently blocking 16 of 35 roster bots. Upload the file to your site root as /robots.txt — one file per host.
How it works
- 1
Pick a preset
Start from 'Block training, allow AI search' (recommended for brands that want AI visibility), welcome everything, or block everything.
- 2
Fine-tune per bot
Toggle individual crawlers on or off. Each row shows the operator, what the bot does, and whether it actually respects robots.txt.
- 3
Merge and install
Optionally paste your existing robots.txt to append the AI rules, then copy or download the file and upload it to your site root.
Blocking training and blocking AI search are different decisions
Most "block AI bots" advice treats AI crawlers as one thing. They are at least four: training crawlers (GPTBot, ClaudeBot, CCBot) collect content to train future models; search indexers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) build the indexes AI assistants cite from; user-triggered fetchers (ChatGPT-User, Claude-User) retrieve a page because a human just asked about it; and control tokens (Google-Extended, Applebot-Extended) are not crawlers at all — they are robots.txt-only switches governing how already-crawled data may be used.
Blocking training is a licensing choice: it says "don't train models on my content" and costs you nothing in AI search visibility today. Blocking search indexers is a visibility choice: OpenAI states plainly that OAI-SearchBot access is required to appear in ChatGPT search results. A brand that wants to be recommended by AI assistants but not folded into training corpora blocks the first group and allows the second — which is exactly what the recommended preset does.
The Google-Extended misconception
The single most common mistake in AI robots.txt files: blocking Google-Extended in the belief that it removes you from AI Overviews. It doesn't. AI Overviews and AI Mode are built on the ordinary Google Search index crawled by ordinary Googlebot; Google-Extended only controls whether that data feeds Gemini training and grounding. It also never fetches anything — you will never see it in your logs, because it is a token other Google crawlers check, not a bot. Apple's Applebot-Extended works the same way: blocking it opts you out of Apple Intelligence training while keeping you in Siri and Spotlight.
robots.txt is a request, not a firewall
Every rule this tool writes is voluntary. The major operators — OpenAI, Anthropic, Google, Apple, Mistral, DuckDuckGo — have good compliance records and honor these rules. Others don't: Bytespider (ByteDance) and DeepSeekBot are widely documented ignoring robots.txt, and xAI publishes no crawler documentation at all. When you block a non-compliant bot here, the generated file marks it with a comment saying so, because the honest version is: for those bots this is a formal, legally useful signal, and the enforcement layer is your WAF or bot management, not this file.
Once your rules are live, two follow-ups are worth a minute. First, verify what your robots.txt actually allows with the AI Crawler Access Checker — the matching rules are less obvious than they look. Second, remember that crawler access only determines whether AI can read you. Whether AI assistants actually mention you when your customers ask is a different question — that's what the free AI Visibility Checker measures.
Frequently asked questions
- Does blocking Google-Extended remove my site from AI Overviews?
- No. Google-Extended only governs whether Googlebot-crawled data can be used for Gemini training and grounding. AI Overviews and AI Mode ride the normal Google Search index, so the only way out of them is out of Google Search itself. Google-Extended is a robots.txt token, not a crawler — it never appears in your server logs.
- Will AI bots actually obey my robots.txt?
- The documented crawlers from OpenAI, Anthropic, Google, Apple, and Common Crawl have good compliance records. But robots.txt is a voluntary convention, not enforcement — Bytespider and DeepSeekBot are widely reported to ignore it. For those, the rules this tool generates are a formal signal; an actual block needs WAF or bot-management rules at your CDN or firewall.
- What is the difference between GPTBot and OAI-SearchBot?
- GPTBot collects content for training OpenAI's models; blocking it is an opt-out from training and does not affect ChatGPT search. OAI-SearchBot builds the index ChatGPT search cites from — OpenAI says access is required to appear in ChatGPT search results. That is why the recommended preset blocks GPTBot but allows OAI-SearchBot.
- Should I block AI training crawlers?
- It is a licensing and positioning choice, not an SEO one. Blocking training bots stops future model training on your content (from compliant operators, going forward) but has no effect on whether AI search engines cite you today. Many publishers block training and allow search; many SaaS brands allow everything for maximum exposure. There is no universally right answer, which is why the presets exist.
- What are user-triggered fetchers like ChatGPT-User or Perplexity-User?
- They fetch a page live because a human just asked an assistant about it — the closest thing AI has to a visitor. Blocking them usually hurts you, because the assistant answers about your site without your site. Several operators also state robots.txt may not apply to user-initiated fetches, and Perplexity's docs say Perplexity-User generally ignores it.
- Where does the generated file go?
- At your site root: https://yoursite.com/robots.txt. There is only one robots.txt per host, so if you already have one, paste it into the merge box and this tool appends the AI rules below your existing rules instead of replacing them.
Related free tools
AI Crawler Access Checker
Check which AI bots your robots.txt allows or blocks — ChatGPT, Claude, Perplexity, Gemini and more, with the exact matching rule.
llms.txt Generator
Generate a well-formed llms.txt from your site's real structure and metadata — no AI-invented descriptions. Crawl, review, download.
XML Sitemap Generator
Crawl your site and download a clean sitemap.xml with truthful lastmod dates — the discovery file AI and search crawlers actually use.
Now find out what AI models actually say about your brand
These tools make your site readable to AI. The free AI Visibility Checker probes grounded AI models with live web search and shows the real answers, your score, and who wins instead of you — no login required.
Run the free AI Visibility Checker