An AI crawler shows up with a budget, not patience. llms.txt is a curated map at your site root: what the site is, what matters, and in what order. Crawler support is still uneven, and this demo shows the proposed mechanism, not a guarantee. But the file takes an hour to write, and the worst case is a text file nobody reads.
The model. Every fetch costs tokens. With no file, the crawler fetches full HTML pages breadth first, in nav order. With a file, it reads /llms.txt, then fetches the markdown twins it lists, in listed order. Either way it stops at the first page that does not fit what is left of the budget.
The answer. The question is answered by quoting the best-matching passage among the pages this run actually ingested (keyword overlap, earliest page wins ties). It is graded on three facts from /docs/rate-limits: 600 requests per minute, bursts up to 100, and the 429 response.
Fernweh Analytics is fictional. Page sizes are fixed per page; the file's own cost is its length divided by four.