llms.txt, explained for developers

What llms.txt is, the exact file format, how it differs from robots.txt and sitemaps, who adopted it, and how to write one that AI assistants actually use.

If robots.txt is how a website says "here is what you may read," llms.txt is how it says "here is what is worth reading, and in what order." It is one of the simplest proposals to come out of the AI-and-the-web moment, it is spreading fast through developer documentation, and it takes about an hour to do well. Here is the whole picture.

What llms.txt is

llms.txt is a proposed standard: a Markdown file served at the root of your site (/llms.txt) that gives language models a curated map of your content. The problem it solves is context: models cannot ingest your whole site, and HTML pages drown the useful part in navigation, scripts, and layout. llms.txt hands a model the short version: what this site is, what matters most, and direct links to clean text.

The proposal has a second half people miss: pages worth reading should also exist as plain Markdown at the same URL with .md appended. The HTML page stays for people; the .md twin serves machines.

The exact format

The spec is deliberately rigid so both models and ordinary parsers can rely on it. In order:

1. An H1 with the site or project name. This is the only required section.
2. A blockquote summarizing the project, carrying the key facts needed to interpret everything below.
3. Optional prose sections with details, anything except headings.
4. Zero or more H2 sections containing file lists: Markdown links, each optionally followed by a colon and a note.
5. An optional section literally titled Optional, with a defined meaning: these links may be skipped when context is tight.

A minimal example in the spec's shape:

# Acme Analytics

> Acme is a self-hosted product analytics server with a REST API
> and SDKs for six languages.

## Docs

- [Quickstart](https://acme.dev/docs/quickstart.md): install to first event in ten minutes
- [REST API](https://acme.dev/docs/api.md): every endpoint with request and response examples

## Optional

- [Changelog](https://acme.dev/changelog.md)

That is the whole standard. No XML, no schema registry, no tooling required to start.

llms.txt vs robots.txt vs sitemap.xml

The three root files answer three different questions, and none replaces another.

• robots.txt answers "what may automated clients fetch?" It is access policy, scoped to crawlers, and in 2026 it is doing heavy lifting we covered in how robots.txt applies to AI agents.
• sitemap.xml answers "what exists?" It enumerates everything for search indexing, without curation, and in aggregate is far too large for a context window.
• llms.txt answers "what should a model read first, and where is the clean version?" It is editorial, not exhaustive: a curated overview meant for inference time, when a user is actively asking about your product, rather than for training.

They compose: robots.txt grants access, sitemap.xml enumerates, llms.txt curates.

Who is using it

Adoption started where the pain is sharpest: developer documentation, where users constantly paste docs into assistants and coding agents. The FastHTML project maintains the reference implementation, publishing both its llms.txt and .md twins of every docs page, and the nbdev documentation ecosystem generates .md versions of all pages by default. Public directories like llmstxt.site track sites that publish the file.

Honesty requires the caveat: llms.txt is a proposal, not an IETF standard, and no major assistant has committed to fetching it on every request. Today its reliable consumers are coding tools and agents whose users point them at it deliberately. That is also why it costs so little to adopt: the file is useful the first time one user's agent reads it, and its value only grows if consumption becomes automatic.

Writing a good one

The spec's own guidance, plus what we see reading the web at index scale:

1. Curate ruthlessly. llms.txt is not a second sitemap. Ten links a model should actually follow beat two hundred it cannot.
2. Make the blockquote load-bearing. One or two sentences that let a model answer "what is this and who is it for" without following a single link.
3. Annotate the links. The : note after each link is what lets a model pick the right one instead of fetching all of them.
4. Serve real Markdown at the targets. The file is a map; the destinations should be clean text, not links back into HTML soup.
5. Use the Optional section honestly. Put genuinely secondary material there, so tight-context readers know what to skip.
6. Keep it current. A stale llms.txt misleads the exact readers who trusted you enough to use it.

Where an index fits in

llms.txt is each site solving machine readability for itself, one file at a time, for the sites that opt in. An AI-readable index solves the same problem from the other side, for every public page at once: Lyrenth crawls the web and serves every URL as a clean, structured AIDocument, whether or not the site ever published a Markdown twin. The two are complementary, and we read them in that spirit: site-provided structure is a signal of a well-maintained machine-readable site, and machine readability is exactly what our AI Readiness score measures for site owners.

If you publish documentation, do both: write the llms.txt (it is an hour well spent for every agent your users point at you), and check your readiness score to see your site the way machine readers see it. And if you are on the consuming side, building the agent rather than the docs, the index already holds the clean version of more than 2 billion pages: the quickstart is one API call.