What is llms.txt? A curated map for AI crawlers

What you're seeing

The left side is the site as a link graph. The home page sits at the root, with marketing pages, pricing, four blog posts, two legal pages, careers, and four documentation pages. The four docs pages carry white rings. They are the key content, the pages that answer the developer questions people actually ask. The red dot is the crawler. Every fetch costs tokens, and the crawler stops at the first page that no longer fits in its budget.

In the no-file run, the crawler does what crawlers do by default. It fetches the home page and walks the nav links breadth first, pulling full HTML pages. At the default budget of 8,000 tokens, it gets through the home page, features, customers, pricing, and one blog post, then runs out at /about (1,000 tokens needed, 200 left). It spent 7,800 tokens and reached none of the docs.

Then the question arrives: "What are Fernweh's API rate limits?" The answer can only quote what the crawler ingested, and the closest thing it has is the pricing page: The Growth plan includes full API access with generous rate limits. The demo grades that as VAGUE, because it contains none of the three real facts.

In the llms.txt run, the crawler reads the file first. It costs 132 tokens and lists the four docs pages. Then the crawler jumps straight to their markdown versions in listed order. Total spend is 2,932 tokens, 95% of it on key pages, and all four docs pages are in. The same question now gets the rate-limits page quoted back with all three facts, cited to /docs/rate-limits.

The scoreboard keeps both runs side by side at the current budget. Two settings make the point harder to argue with:

• Budget 16,000, no file. Even with twice the budget, nav order buries the docs. The crawler reaches only /docs/quickstart (25% key coverage) before the blog and company pages use up the rest, and the answer still cites the pricing page.
• Budget 4,000, docs-only file. Half the default budget still covers all four docs pages, because the file plus their markdown versions cost 2,932 tokens.

Now open the editor and press All 16. The file lists every page, written in the site's nav order the way most generators write it. At 8,000 tokens, coverage drops to 0%, the same as having no file at all, because the crawler spends the budget on the product and blog entries listed above the docs. A file that lists everything is a sitemap. The value is in the curation.

What is llms.txt?

llms.txt is a proposed convention for a markdown file served at /llms.txt that gives language models a short, curated guide to a website. Jeremy Howard of Answer.AI proposed it in September 2024, and the spec lives at llmstxt.org. The proposal's own framing is blunt: "Agents now use websites constantly. But web pages are built for people."

A web page carries navigation, scripts, cookie banners, and layout markup. A model reading it pays for all of that in its context window. llms.txt gives it a shortcut: one small file that says what the site is, which pages matter, and where clean versions of those pages live. The spec says it expected the file "would mainly be useful for inference rather than training", meaning an agent reading your site to answer a question right now, not a model being trained on it.

The llms.txt format, line by line

The file is ordinary markdown with a fixed order of parts. Only the first one is required.

1. An H1 with the site or project name. This is the only required line.
2. A blockquote summary. One or two sentences with the key facts a reader needs to make sense of everything below it.
3. Optional detail. Paragraphs or lists (anything but headings) that explain the project or how to read the linked files.
4. H2 sections of links. Each item is a markdown link, optionally followed by a colon and a one-line note: - [Rate limits](/docs/rate-limits.md): Per-key request limits
5. An "Optional" section, last. Links an agent can skip when it needs a shorter context.

The spec also suggests a markdown twin for each page worth reading, at the same URL with .md added. That is why the demo's file run is so much cheaper per page: it fetches /docs/rate-limits.md (400 tokens) instead of the full HTML page (1,100 tokens). As of the spec's August 2026 revision, the file can also live at a subpath, where it covers only the pages under that path.

llms.txt example

This is the file the demo builds when only the four docs pages are checked. It is short on purpose.

# Fernweh Analytics

> Product analytics for teams that ship weekly: event tracking, funnels, and retention cohorts, with a REST API for all of it.

Each link points at the markdown twin of the page.

## Docs

- [Quickstart](/docs/quickstart.md): Install the SDK and send a first event
- [API reference](/docs/api-reference.md): Every REST endpoint, with request examples
- [Rate limits](/docs/rate-limits.md): Per-key request limits and the 429 response
- [Webhooks](/docs/webhooks.md): Delivery, retries, and signature checks

And here is the start of the real file this site serves, trimmed. It is generated from the posts collection on every request, so a new article can never be missing from it. The how this site is built post covers the setup, including the markdown twins and the companion llms-full.txt that holds the full text of every article.

# Pro Trailblazer

> Practical guides, products, and recommended solutions for people doing the real work.

Pro Trailblazer publishes three types of content: in-depth articles with interactive demos, Pro Trailblazer products (sold via Polar), and recommended third-party solutions (affiliate). ...

## Reference

- [Terms Dictionary](https://protrailblazer.com/terms): Filterable glossary of industry terms and concepts, with schema.org DefinedTermSet markup.
- [Articles](https://protrailblazer.com/posts): All published guides and essays.
- [Full content](https://protrailblazer.com/llms-full.txt): Clean full text of every published article in one file.

Every article also has a markdown twin at the same URL with a `.md` suffix (for example `/posts/retrieval-augmented-generation.md`).

## Articles

- [Context window: what an LLM sees, and what it forgets](https://protrailblazer.com/posts/context-window-explained/): Watch two LLM context windows, 2k and 32k tokens, slide through the same text and drop exactly the tokens that scroll off the back of the smaller one.
- ...

Fair warning: this site's file lists every article, which is the pattern the demo punishes at small budgets. The trade is different here. Nearly every page on this site is an article meant to be read, and the reference links come first. A SaaS site with a few key docs pages and a lot of marketing pages should be much pickier.

llms.txt vs robots.txt

They answer different questions, so a site can use both without conflict.

• robots.txt controls access. It says which crawlers may fetch which paths. It is a real standard (RFC 9309), and major crawlers honor it. It is the only one of the two that can keep an AI crawler out.
• llms.txt gives guidance. It says what is worth reading and in what order. It is a proposal, it enforces nothing, and a crawler is free to ignore it.
• sitemap.xml lists everything. The llms.txt spec draws this line itself: "While sitemaps list all pages for search engines, llms.txt offers a curated overview for LLMs."

This is the robots.txt pattern this site uses to allow the major AI crawlers by name (trimmed):

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /

Sitemap: https://protrailblazer.com/sitemap-index.xml

Never list a page in llms.txt that robots.txt blocks. The two files should agree on what is public.

Do AI crawlers use llms.txt?

Mostly no, as of October 31, 2026. Here is what the primary sources say:

• Google says you do not need it. Google's AI optimization guide (last updated July 10, 2026) says you can ignore tactics like "creating unnecessary AI text files (like llms.txt)" for Google Search, and adds that it is fine to keep them for other services.
• Chrome checks it, lightly. Lighthouse added an experimental agentic browsing category with an llms.txt audit. A 404 counts as not applicable, so the audit only flags a file that errors when fetched. It does not measure ranking or citations.
• No major AI crawler documents support. I could not find documentation from OpenAI, Anthropic, or Perplexity saying their crawlers use llms.txt to decide what to fetch or cite.

Where llms.txt does get read is on demand. Coding agents and chat assistants that can fetch URLs will read an llms.txt when a person points them at a docs site, and that is the use the spec was written for. If your audience is developers, that alone can justify the hour it takes.

How to create an llms.txt file

1. Pick the pages that answer real questions. Five to twenty is a good range for most sites. Docs, pricing, and your best reference pages, not every blog post.
2. Order them by importance. The demo's overstuffing result is really an ordering result: every entry is read before the entries below it.
3. Write one plain line per link. Say what is on the page ("Per-key request limits and the 429 response"), not marketing copy. The file is itself something a model reads and can quote.
4. Put the rest under "## Optional". Legal pages, older posts, and anything nice to have go last.
5. Serve markdown twins if you can. Clean .md versions are what make each fetch cheap.
6. Generate it from your content. A hand-written file goes stale the first time someone publishes. This site replaced its hand-maintained file with a generated one for exactly that reason.
7. Check it. Fetch /llms.txt yourself, confirm it returns 200 as plain text, and run the Lighthouse audit.

What the demo simplifies

• The budget is a fixed token count. Real crawlers and agents are limited by requests, time, and how much fits in a context window, and they may skip a page that does not fit and keep going instead of stopping.
• The answer comes from keyword overlap between the question and each ingested passage. Real systems use embeddings and a model to write the answer. The retrieval-augmented generation demo shows that pipeline. The lesson is the same: the model can only cite what it retrieved.
• Page sizes are invented, though the gap between HTML and markdown versions is realistic.
• With a file, the crawler reads only what the file lists. A real crawler might keep crawling afterward with the budget it has left.

Frequently asked questions

Key takeaways

• llms.txt is a curated markdown map at /llms.txt: an H1, a one-line summary, and sections of links with short notes, with "Optional" last.
• robots.txt decides where a crawler may go. llms.txt suggests what is worth reading. Only robots.txt can block anything.
• A budget-limited reader that follows nav order can spend all of its budget before it reaches the docs. In the demo, doubling the budget to 16,000 tokens still got only 25% of the key pages.
• Curation is the whole point. A file that lists every page in nav order did no better than having no file at all.
• As of October 2026, Google says it does not use llms.txt, and no major AI crawler documents support. Agents that fetch docs on request are the real audience.