llms.txt is a plain Markdown file, served from the root of a website, that gives large language models a curated summary of the site’s most important content. Jeremy Howard of Answer.AI proposed the standard in September 2024. No major AI engine has confirmed it reads the file, so treat it as a cheap experiment, not a proven ranking signal.
The pitch behind llms.txt is straightforward. Web pages are built for browsers: they carry navigation, scripts, adverts and boilerplate that waste a model’s context window. A single Markdown file that lists your canonical resources with one line descriptions lets a model find the right page without parsing any of that clutter.
The catch is adoption. Thousands of sites now publish the file, including Anthropic, Cloudflare and Zapier, yet the crawlers that matter have stayed silent. This guide sets out the format, the evidence for and against, and the fastest way to ship one, so you can make the call on facts rather than hype.
What is llms.txt and who created it?
llms.txt is a proposed web standard created by Jeremy Howard, co-founder of Answer.AI, in September 2024. It is a UTF-8 Markdown file served at /llms.txt that names the site, summarises what it does and links to its most useful pages, so language models can retrieve relevant material without parsing full HTML.
The full specification lives at llmstxt.org. The format is deliberately strict so that both humans and parsers can rely on it:
- An H1 with the site or project name. This is the only required element.
- A blockquote immediately below it containing a short summary.
- Optional free-form Markdown paragraphs with context the model should know.
- H2 sections containing link lists, where each entry follows the pattern title: description.
- An H2 named Optional for secondary links that an agent can skip when its context window is tight.
The proposal also recommends publishing a Markdown twin of each important page at the same URL with .md appended, so an agent that discovers a page through llms.txt can fetch a clean text version of it. Documentation platform Mintlify added automatic llms.txt generation in November 2024, which pushed the file onto thousands of developer documentation sites almost overnight.
What is llms-full.txt?
llms-full.txt is a companion convention that concatenates the complete Markdown content of every listed page into a single file. It suits tools that ingest an entire documentation set in one request. The trade-off is size: large sites produce llms-full.txt files of several hundred thousand tokens, which exceeds many models’ context windows entirely.
How is llms.txt different from robots.txt and sitemap.xml?
robots.txt controls which crawlers may access which URLs, sitemap.xml lists every indexable URL to aid discovery, and llms.txt curates and describes your best content for language models. llms.txt grants no permissions and blocks nothing. A crawler that ignores it loses nothing, while one that reads it gains an annotated map of your site.
| File | Core job | Primary audience | Standing |
|---|---|---|---|
| robots.txt | Access control: allow or block crawling by user agent | All compliant crawlers | De facto standard since 1994, formalised in RFC 9309 in 2022 |
| sitemap.xml | URL discovery: list every page you want indexed | Search engine crawlers | Jointly adopted by Google, Yahoo and Microsoft in 2006 |
| llms.txt | Curation: describe and prioritise key content for AI | LLMs and AI agents | Proposal from September 2024 with no confirmed engine support |
One practical consequence matters more than the taxonomy. llms.txt cannot stop AI companies training on your content. If that is your goal, add disallow rules for GPTBot, ClaudeBot and Google-Extended to robots.txt, because those crawlers document robots.txt compliance and say nothing about llms.txt.
Do AI engines actually read llms.txt?
No major AI engine has confirmed that it uses llms.txt. Google Search Advocate John Mueller stated in June 2025 that, to his knowledge, no AI service fetches the file, and he compared it to the keywords meta tag. OpenAI, Anthropic and Perplexity all document their crawler behaviour, and none lists llms.txt support.
The server log evidence points the same way. Independent analyses of sites publishing the file through 2025 found sporadic fetches at best, usually from an agent following a link a human supplied rather than from systematic crawling.
Two nuances keep the file from being a dead letter:
- Training crawls and live retrieval are different pipelines. The big training crawlers ignore llms.txt today, but agentic tools that browse on demand, such as coding assistants fetching framework documentation, do retrieve it when a user or a tool configuration points at it. That is why adoption is strongest among developer documentation sites.
- The AI companies hedge their own bets. Anthropic publishes llms.txt and llms-full.txt for its own documentation. Publishing is not the same as consuming, but it signals that the proposal is taken seriously inside those organisations.
The honest summary for 2026: implementing llms.txt is a 30 minute bet on a standard that might get adopted, not a tactic with measurable returns today.
How do you create an llms.txt file?
Create a UTF-8 Markdown file named llms.txt, give it an H1 with your site name, a one paragraph blockquote summary, then H2 sections of links to your key pages with one line descriptions, and serve it from your domain root. Most sites can publish a useful version in under an hour.
A minimal example:
# Example Site
> Example Site helps founders automate organic growth with AI driven SEO and answer engine optimisation.
## Product
- [How it works](https://example.com/how-it-works): The autonomous content pipeline, step by step.
- [Pricing](https://example.com/pricing): Plans and what each includes.
## Guides
- [Blog](https://example.com/blog): Practical articles on AI search visibility.
## Optional
- [Press kit](https://example.com/press): Logos and company facts.Five rules keep the file useful:
- Curate, do not dump. llms.txt is not a sitemap. List the 10 to 50 pages that best answer the questions your audience asks, not every URL you own.
- Write the descriptions for retrieval. Each one line description is what a model reads when deciding which link to fetch, so state what the page answers in plain subject-verb-object prose.
- Keep it current. A stale llms.txt that points at renamed or deleted URLs is worse than none. Regenerate it as part of your publishing pipeline, not by hand.
- Use the Optional section honestly. Anything a model could skip without harming its answer belongs there.
- Serve it as plain Markdown. No HTML, no redirect to a rendered page, and a text/plain or text/markdown content type.
Does llms.txt improve answer engine optimisation?
Not on its own. AI citations in 2026 come from pages that answer questions directly in the first 50 words, use question-led headings, load fast and expose clean, crawlable HTML. The presence of an llms.txt file changes none of that, so treat it as a minor output of a sound AEO process, not a shortcut to visibility.
What demonstrably earns citations across ChatGPT, Perplexity, Gemini and Google AI Overviews:
- Answer-first structure: open the page, and every section, with a 40 to 60 word direct answer an engine can lift verbatim.
- Question-led headings: phrase H2s the way people phrase prompts, because retrieval matches questions to questions.
- Semantic triples: state facts as subject-verb-object sentences with concrete numbers, dates and named sources.
- Structured data: Article and FAQPage schema make extraction cheaper for every engine that parses it.
- Internal linking: dense, descriptive links tell crawlers which page is canonical for which question.
- Freshness: engines prefer recently updated sources for time-sensitive queries.
This is the layer where effort compounds. rankloop automates it: the platform plans question-led topics, writes answer-first articles, builds the internal links between them and keeps them fresh, so every new page strengthens the ones before it. The full pipeline is documented at how rankloop works, and the organic growth it produces is published openly on the results page. An llms.txt file generated automatically from that kind of content library is genuinely useful to an agent; one bolted onto thin content is a map to nowhere.
Frequently Asked Questions
Where should the llms.txt file be located?
llms.txt belongs at the root of your domain, for example https://example.com/llms.txt, served as plain Markdown text with UTF-8 encoding. Subdirectory placements are not part of the proposal, so tools will not look for them. If you also publish llms-full.txt, the full-content companion file, it sits at the domain root alongside llms.txt.
Is llms.txt an official web standard?
No. llms.txt is a community proposal published by Jeremy Howard of Answer.AI in September 2024, with its specification maintained at llmstxt.org. No standards body such as the IETF or W3C has ratified it, and no major AI provider has committed to reading it. Adoption so far consists of voluntary publishing by website owners, strongest among developer documentation platforms.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a curated index: a site summary plus annotated links to key pages, designed to stay small. llms-full.txt concatenates the full Markdown content of all of those pages into one large file so a tool can ingest everything in a single request. Small sites can publish both; very large sites often skip llms-full.txt because the file exceeds model context windows.
Does Google support llms.txt?
No. Google has not announced any support for llms.txt, and Google Search Advocate John Mueller said in June 2025 that no AI system he knew of was using the file, likening it to the keywords meta tag. Google’s AI products rely on standard crawling, and its AI training opt-out is the Google-Extended token in robots.txt, not llms.txt.
Can llms.txt stop AI companies training on my content?
No. llms.txt has no access control function; it only describes and links content. To restrict AI training crawlers, add disallow rules to robots.txt for the documented user agents, such as GPTBot for OpenAI, ClaudeBot for Anthropic and Google-Extended for Google’s AI training. Those companies state they respect robots.txt, and none has made any commitment regarding llms.txt.
How long should an llms.txt file be?
Keep llms.txt small enough to sit comfortably inside a model’s working context, which in practice means a short summary plus roughly 10 to 50 annotated links. The specification sets no hard limit, but the file’s value is curation: if it starts to approach the size of a sitemap, move secondary links into the Optional section or cut them entirely.
Should you publish an llms.txt file?
Publish one if it costs you less than an hour, and automate its generation so it never goes stale. The downside is zero and the upside is a free position if AI engines adopt the standard. Then put your real effort where the evidence already sits: answer-first, interlinked content that AI engines cite today. rankloop builds exactly that, autonomously, article by article. Start with rankloop and get a content engine that compounds whether or not llms.txt ever takes off.