whereamimentioned
Get started
Free toolsChecker

AI Crawler Access Checker

Enter your domain and we'll fetch your robots.txt, evaluate it exactly the way crawlers do (RFC 9309 rules), and show which AI bots can see your site — with the exact rule that decides each verdict.

How it works

  1. 1

    Enter your domain

    We fetch your site's robots.txt file — nothing else.

  2. 2

    We evaluate every AI bot

    Each user-agent is matched to its robots.txt group using the same precedence rules real crawlers use: most-specific group wins, longest path match, allow beats disallow on ties.

  3. 3

    Read the verdicts

    See who's allowed, who's blocked, which rule did it, and what each verdict means for your AI visibility.

Why AI crawler access decides your AI visibility

Every AI assistant that cites sources — ChatGPT search, Perplexity, Claude, Gemini — depends on crawlers reaching your site. If your robots.txt blocks OAI-SearchBot, you don't appear in ChatGPT search results; OpenAI says so explicitly. Many sites blocked AI bots wholesale in 2023–2024 (often via a CDN default or a copy-pasted block list) and are now invisible to AI search without knowing it. This checker exists to catch exactly that.

The categories matter more than the individual bots. Training crawlers (GPTBot, ClaudeBot, CCBot…) collect data for future models — blocking them is a licensing/positioning decision. Search indexers (OAI-SearchBot, PerplexityBot, Claude-SearchBot…) build the indexes AI answers cite from — blocking them removes you from those answers. User-triggered fetchers (ChatGPT-User, Perplexity-User…) fetch your pages live because a human asked about you — blocking them means the assistant answers about your brand without your site.

How the verdicts are computed

We parse your robots.txt with the same semantics as RFC 9309 and Google's published spec: a crawler follows only its single most-specific matching user-agent group (a named GPTBot group makes GPTBot ignore User-agent: * rules entirely), the longest matching path rule wins within a group, and Allow beats Disallow on exact ties. Wildcards (*, $) are supported, paths are case-sensitive, and files over 500 KiB are truncated the way real parsers truncate them. Each verdict shows the exact line that decided it, so you can fix rules with confidence.

One honest caveat: robots.txt is a request, not a fence. Where a bot has a documented history of ignoring it, we label the verdict "asked not to crawl" — enforcement against those requires WAF-level blocking.

Allowed is necessary — mentioned is the goal

Crawler access is step zero: it determines whether AI engines can read you, not whether they recommend you. Once your access is clean, the question becomes what AI models actually say when customers ask about your category — that's what our free AI Visibility Checker measures, with live web search and the raw answers shown.

Frequently asked questions

Which AI crawlers does this checker cover?
Around 30 user-agents across OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity, Google (Google-Extended, GoogleOther), Apple, Meta, Amazon, Mistral, ByteDance, Common Crawl, xAI and more — each labeled by purpose: training, AI search indexing, or user-triggered fetching.
If I block a bot in robots.txt, is it actually blocked?
Only if the bot chooses to comply. Robots.txt is a voluntary protocol — most major operators respect it, but some (Bytespider, DeepSeek, and reportedly Grok-related traffic) ignore it, and Perplexity's user-triggered fetcher openly states it generally ignores robots.txt. For non-compliant bots this checker says 'asked not to crawl' instead of 'blocked' — actually stopping them requires WAF or bot-management rules.
Does blocking Google-Extended remove me from Google's AI Overviews?
No — this is the most common misconception. Google-Extended only controls whether your content trains and grounds Gemini. AI Overviews and AI Mode are built from the normal Google Search index crawled by Googlebot, so the only way out of them is leaving Google Search or restricting snippets.
What happens if my site has no robots.txt at all?
A missing robots.txt (HTTP 404) means every crawler is allowed everywhere — a perfectly valid choice if you want maximum AI visibility. A 5xx error is different: compliant crawlers treat a failing robots.txt as 'disallow everything' until it recovers.
Why would blocking AI crawlers hurt my business?
Blocking AI search indexers like OAI-SearchBot or PerplexityBot removes you from the answers those assistants give — OpenAI explicitly says OAI-SearchBot access is required to appear in ChatGPT search. Blocking user-triggered fetchers stops assistants from reading your site when a customer asks about you specifically. Blocking training crawlers, by contrast, is a licensing choice that doesn't affect today's AI search visibility.

Related free tools

Generator

AI Crawler robots.txt Generator

Build robots.txt rules for GPTBot, ClaudeBot, PerplexityBot and 25+ AI crawlers — allow AI search, block training, or go custom.

Generator

llms.txt Generator

Generate a well-formed llms.txt from your site's real structure and metadata — no AI-invented descriptions. Crawl, review, download.

Generator

XML Sitemap Generator

Crawl your site and download a clean sitemap.xml with truthful lastmod dates — the discovery file AI and search crawlers actually use.

Now find out what AI models actually say about your brand

These tools make your site readable to AI. The free AI Visibility Checker probes grounded AI models with live web search and shows the real answers, your score, and who wins instead of you — no login required.

Run the free AI Visibility Checker