What is llms.txt?
It’s a plain-text file, written in Markdown and hosted at the root of the domain, that acts as a curated index of the site for language models. It contains the organisation’s name, a neutral summary of what it does and a list of the most relevant pages with a short description of each.
The proposal came out in 2024 from the AI research community and rests on a simple observation: when a model retrieves a web page, much of what it receives is navigation, footers, cookie notices and markup. The useful content arrives diluted. A Markdown file delivers it clean.
How does it differ from robots.txt?
They serve opposite, complementary functions. It’s worth not confusing them, because that’s the most common mistake.
robots.txt |
llms.txt |
|
|---|---|---|
| Function | Control access | Guide discovery |
| Nature | Binding | Advisory |
| Syntax | Allow / Disallow directives |
Markdown with summaries |
| Audience | Crawlers | Language models |
robots.txt dictates permissions: what may be visited and what may not. llms.txt
restricts nothing; it just points to what’s worth reading. One is a gate, the other
a signpost.
Of the two, the one that can really ruin your visibility is robots.txt. If your
firewall or configuration blocks GPTBot, ClaudeBot, PerplexityBot or
Google-Extended, no llms.txt will make up for it: you’ll be publishing an index
for readers you’ve shut the door on.
Does anyone respect it?
Here we have to be honest, because there’s a lot of poorly calibrated enthusiasm around this file.
llms.txt is not a standard approved by any body, and as of today the major
model providers haven’t publicly confirmed that they consume it systematically.
There’s growing adoption in technical documentation and some retrieval tools make
use of it, but anyone promising that publishing an llms.txt will improve your
citations is selling a certainty no one has.
That said, the maths still favours it. Writing it takes an afternoon, has no contraindications, and the exercise of deciding which are your twenty important pages and summarising each in fifteen words usually reveals more about a site’s architecture than most audits.
How to write a decent one
- One
H1with the organisation’s name. - A block quote summarising the business in two sentences, with no sales language. “Web engineering studio” works; “leaders in innovative digital solutions” is noise that degrades the signal.
H2sections grouping ten to twenty links to the core pages, each with a context description of about fifteen words.- Nothing else. No menus, no legal notices, no full blog listing.
If the site is small, you can add an llms-full.txt consolidating all the text into
a single flattened document, meant for models with large context windows.
Our own file
This site’s is at /en/llms.txt and is generated on every build from the same content collections that feed the pages. Writing it by hand guarantees it will lie within three months.