llms.txt: What It Is and How to Add One
llms.txt is a plain-text site map for AI assistants. What goes in it, how it differs from robots.txt, how to add one in an afternoon, and a live reference implementation.
By John Patterson, Founder · Published
The short answer
llms.txt is a markdown file at your domain root (yoursite.com/llms.txt) that gives AI assistants a curated map of your site: what you do, and which pages matter, with one-line descriptions. Where robots.txt tells crawlers what they may access, llms.txt tells them what's worth reading: it's a table of contents written for machines. The format (proposed by Jeremy Howard in 2024) is deliberately simple: an H1 with your site name, a blockquote summary, then sections of markdown links with descriptions. A companion file, llms-full.txt, can carry full page content for direct LLM consumption. Ours is live at ahoylighthouse.com/llms.txt and serves as the reference implementation for this guide; you can build yours in an afternoon.
What problem does llms.txt solve?
An AI agent landing on your site has a context budget. Crawling every page to figure out what you do burns tokens on navigation chrome, footers, and duplicate content. llms.txt hands the agent a curated index instead: here's who we are, here are the pages that answer real questions, skip the rest. For retrieval-augmented answers, that curation nudges engines toward your best content rather than whatever page happened to get fetched.
What's the format?
Plain markdown, served as text/plain at /llms.txt:
# Your Company: one-line positioning
> Two or three sentences on what you do and who it's for.
## Pages
- [Page title](https://yoursite.com/page): One line on what this page answers.
- [Another page](https://yoursite.com/other): Same idea.
## Blog Posts
- [Post title](https://yoursite.com/blog/slug): What the post covers.
Rules of thumb: absolute URLs, one honest descriptive line per link (the descriptions are what agents read to decide relevance), most important content first, and keep it maintained: a stale llms.txt pointing at dead pages is worse than none.
llms.txt vs llms-full.txt vs robots.txt
| File | Job | Audience |
|---|---|---|
| robots.txt | Access control: what crawlers may fetch | All crawlers |
| llms.txt | Curation: what's worth reading, with descriptions | LLM agents |
| llms-full.txt | Content delivery: full text of key pages in one file | LLM agents with bigger context budgets |
They compose: robots.txt should allow AI crawlers and can point to your sitemap; llms.txt indexes your best pages; llms-full.txt (optional) inlines their content. We serve all three (robots.txt, llms.txt, llms-full.txt) and generate the latter two dynamically so new blog posts appear without manual edits.
How do you add one?
- Write the header: site name, one-line positioning, a blockquote summary an agent could quote verbatim.
- List 10-30 pages grouped into sections (Pages, Guides, Blog), each with a one-line description that says what question the page answers.
- Serve it at /llms.txt with Content-Type: text/plain. A static file works; generating it dynamically (from your CMS or database) keeps it current automatically.
- Reference it: a comment line in robots.txt pointing at it helps agents find it.
- Verify: curl yoursite.com/llms.txt, check it returns 200 as plain text, and read it as a stranger: does it actually explain your site?
Does llms.txt actually matter yet?
Honest answer: adoption by the engines is uneven and the standard is young. OpenAI, Anthropic, and Perplexity crawlers will fetch it, and agent frameworks increasingly look for it, but nobody publishes hard numbers on citation lift from llms.txt alone. The case for doing it anyway: it costs an afternoon, it can't hurt, it improves how coding and research agents (a fast-growing traffic source) understand your site, and it signals machine-readability that correlates with the practices that do measurably matter: crawler access, structure, and freshness. We track AI crawler behavior across our own site and llms.txt is among the most-fetched non-HTML resources.
Keep reading
Frequently asked questions
Where does llms.txt go?
At your domain root: yoursite.com/llms.txt, served as plain text (text/plain). Subdirectory placements aren’t part of the convention and agents won’t look there.
What is llms-full.txt?
A companion file carrying the full text of your key pages in one document, for agents that want content rather than an index. It’s optional; generate it dynamically if your content changes often.
Does Google use llms.txt?
Google hasn’t announced support for llms.txt as a ranking or AI Overview input as of mid-2026. OpenAI, Anthropic, and Perplexity crawlers do fetch it. Treat it as low-cost future-proofing rather than a guaranteed lever.
Should llms.txt list every page on my site?
No, that’s what sitemap.xml is for. llms.txt is a curated shortlist (10-30 links) of the pages that best explain what you do and answer real questions, each with a one-line description.
See how AI describes your brand.
Start with the free snapshot: 100 prompts on all seven engines, with every answer and its sources. No card.