What's the difference between llms.txt and llms-full.txt?

llms.txt is an index — site name, summary, and curated links with one-line descriptions. llms-full.txt is the full content: every page's readable text expanded into one markdown file. The index tells a model where things are; the full file gives it the things themselves.

Is llms-full.txt part of the llms.txt spec?

No. It's a de facto convention popularized by Mintlify and Anthropic around November 2024, not part of the core llmstxt.org proposal. Documentation platforms adopted it because it's genuinely useful for feeding docs to assistants — but there's no standard governing it.

Do AI crawlers fetch llms-full.txt?

Essentially no — the same consumption reality as llms.txt applies: Ahrefs found 97% of llms.txt files get zero AI-crawler requests, and no major engine has confirmed using either file for indexing. The proven use today is you feeding your own site to an assistant: a custom GPT's knowledge file, a Claude Project, or a one-off paste.

Why does the tool show a token estimate?

Because real llms-full.txt files routinely exceed LLM context windows — Anthropic's own is around 481,000 tokens. We estimate tokens (roughly 0.75 tokens per word) so you know whether the file fits the assistant you're feeding it to before you try.

Why were some of my pages skipped?

Two reasons: pages with under 40 words of extractable text are skipped as empty (usually JavaScript-rendered pages or thin index pages), and once the file reaches the 2 MB cap remaining pages are listed as skipped. The per-page table shows exactly which and why.

Free toolsGenerator

llms-full.txt Generator

Enter your site URL and get a single llms-full.txt — every page's readable text as one markdown file, with word and token counts so you know if it fits a context window.

How it works

1
Enter your site URL
We crawl up to 20 pages, sitemap-first, respecting your robots.txt.
2
We extract the readable content
Navigation, headers, footers and scripts are stripped; each page becomes a markdown section with its title, source URL, and main text.
3
Check the size
The stats row shows words and estimated tokens — with a warning when the file exceeds typical context windows.
4
Download and use it
Upload to your site root as /llms-full.txt, or feed it directly to a custom GPT or Claude Project.

Index vs. full content — what llms-full.txt actually is

The llms.txt proposal defines an index: a curated markdown map of your site. llms-full.txt is the maximalist sibling — the entire readable content of your site expanded into one file, each page rendered as a # heading, a Source: line with its URL, and the page's main text. It is not part of the core spec: it's a convention that Mintlify rolled out across its documentation platform and Anthropic adopted for its own docs around November 2024, and other docs platforms followed.

The trade-off between the two files is context. An index costs a model a few hundred tokens but requires it to fetch pages one by one; the full file needs no follow-up fetches but can be enormous — Anthropic's llms-full.txt runs to roughly 481,000 tokens, several times most context windows. That's why this tool reports an estimated token count instead of pretending size doesn't matter: if the number exceeds your assistant's window, it will truncate or refuse, and you're better off knowing before you paste.

The honest use case: feeding your own site to an assistant

Let's be straight about consumption, same as on our llms.txt page: AI crawlers are not fetching these files at any meaningful rate — Ahrefs measured that 97% of published llms.txt files get zero AI-crawler requests, and no major engine has confirmed using them. Anyone selling llms-full.txt as an SEO play is ahead of the evidence.

What the file is genuinely good at, today, is the direction nobody markets: you feeding your own site to an AI. Drop it into a custom GPT's knowledge files, a Claude Project, or a support bot's grounding corpus and you've turned your whole site into a single clean attachment — no scraping, no per-page copy-paste, no HTML noise. Teams use exactly this to build product assistants, brief agencies, audit their own copy, or ask "what does our site actually say about pricing?" against the ground truth.

Because the file is extracted server-side from your HTML, JavaScript-rendered pages with no server-side content come out thin — those are skipped (under 40 words) and listed so you know. The 2 MB cap keeps the output practical; if your site is bigger than that, a curated llms.txt index is usually the better artifact anyway.

Your content, as AI sees it — now check what AI says

Generating this file shows you exactly what a text-only reader can extract from your site — an eye-opening audit by itself. The next question is what AI engines actually say when customers ask about your category. The free AI Visibility Checker probes grounded models with live web search and shows the raw answers, your score, and who gets recommended instead of you.

Frequently asked questions

What's the difference between llms.txt and llms-full.txt?: llms.txt is an index — site name, summary, and curated links with one-line descriptions. llms-full.txt is the full content: every page's readable text expanded into one markdown file. The index tells a model where things are; the full file gives it the things themselves.
Is llms-full.txt part of the llms.txt spec?: No. It's a de facto convention popularized by Mintlify and Anthropic around November 2024, not part of the core llmstxt.org proposal. Documentation platforms adopted it because it's genuinely useful for feeding docs to assistants — but there's no standard governing it.
Do AI crawlers fetch llms-full.txt?: Essentially no — the same consumption reality as llms.txt applies: Ahrefs found 97% of llms.txt files get zero AI-crawler requests, and no major engine has confirmed using either file for indexing. The proven use today is you feeding your own site to an assistant: a custom GPT's knowledge file, a Claude Project, or a one-off paste.
Why does the tool show a token estimate?: Because real llms-full.txt files routinely exceed LLM context windows — Anthropic's own is around 481,000 tokens. We estimate tokens (roughly 0.75 tokens per word) so you know whether the file fits the assistant you're feeding it to before you try.
Why were some of my pages skipped?: Two reasons: pages with under 40 words of extractable text are skipped as empty (usually JavaScript-rendered pages or thin index pages), and once the file reaches the 2 MB cap remaining pages are listed as skipped. The per-page table shows exactly which and why.

Related free tools

Generator

llms.txt Generator

Generate a well-formed llms.txt from your site's real structure and metadata — no AI-invented descriptions. Crawl, review, download.

Checker

llms.txt Validator

Validate your llms.txt against the spec: structure, link format, broken links, and common mistakes — with line-by-line results.

Generator

XML Sitemap Generator

Crawl your site and download a clean sitemap.xml with truthful lastmod dates — the discovery file AI and search crawlers actually use.

Now find out what AI models actually say about your brand

These tools make your site readable to AI. The free AI Visibility Checker probes grounded AI models with live web search and shows the real answers, your score, and who wins instead of you — no login required.

Run the free AI Visibility Checker