Do AI companies actually read llms.txt?

No major provider has committed to it, and there is no public evidence of it being used at scale. It is a proposal from 2024 that has been adopted by a number of documentation sites and by some developer tools. Treat it as cheap and speculative rather than as a requirement.

What is the difference between llms.txt and llms-full.txt?

llms.txt is an index: a short description of the site and a linked list of its important pages, each with a sentence. llms-full.txt is the content itself, every page concatenated into one file. The first is a map, the second is the territory, and they serve different requests.

Is llms.txt the same as robots.txt for AI?

No, and the confusion is worth avoiding. robots.txt controls access and is honoured by real crawlers today. llms.txt controls nothing: it is a courtesy summary. Blocking or allowing an AI crawler is a robots.txt decision, and publishing llms.txt does not change what any bot is permitted to do.

llms.txt: what it is, whether it does anything yet, and how to write one

Most of what is written about llms.txt is either dismissal or hype. The useful version is shorter: it is a markdown file at your root that indexes your own best pages, it costs an afternoon, and no assistant vendor has publicly committed to reading it. All three of those are true at once, and the third one is why this page exists.

The short answer

llms.txt is a proposed markdown file at your site root that lists your important pages with a sentence each, so a model can orient without crawling everything. It is unratified and no major provider has committed to reading it. It is worth an afternoon because it is cheap, and because writing one forces you to say what your site is.

ImportantAudit check · llms.txt

What the proposal actually specifies

llms.txt is a file at /llms.txt written in ordinary markdown, and the format is deliberately tiny. An H1 with the name of the site. An optional blockquote summarising what it is. Then H2 sections, each holding a markdown list of links, where every link may carry a short description after a colon. A section named Optional has a defined meaning: those links can be skipped when a shorter context is needed.

That is the entire specification. It is not a manifest, it has no schema to validate against, and there is no header block or version field. It is closer to a hand-written table of contents for your own substance than to anything machine-negotiated, and the format was chosen so that a language model reading it needs no parser at all.

How settled is this, really

Not very, and you should hold it accordingly. It is a community proposal rather than a ratified standard: there is no working group behind it, no RFC, and, as of this writing, no major assistant vendor has publicly committed to fetching or honouring it. Anyone quoting you traffic figures from llms.txt is inventing them.

What can be said without hand-waving is that the cost is close to zero and the downside is zero. A markdown file at a well-known path breaks nothing, slows nothing, and leaks nothing you have not already published. Weighed against a possibility that costs an afternoon, that is a cheap option to hold rather than a growth lever to pull.

Write one because it is nearly free, not because someone promised it would work.

What a pass requires here

A request to /llms.txt that returns a document rather than your 404 page. The check is site-level, so it runs once for the whole crawl and reports the same answer whichever page you started from. A companion check looks for /llms-full.txt separately, since the two files do different jobs and most sites that have one do not have the other.

One detail worth knowing: a site that returns a soft 404, a 200 response carrying your normal not-found page, will look to a naive fetcher like a file that exists. We compare the response against what the site returns for a path that certainly does not exist, so a soft 404 is reported as missing rather than found.

Check a site for llms.txt

Paste any URL on the site. This runs a single-page audit and reports only the site-level answer: whether /llms.txt and /llms-full.txt return a document.

Check a site for llms.txt and llms-full.txt

No signup required. Each free search audits one page, paste any URL to see it in action.

Why this is rated Important and not Critical

Because nothing breaks without it. A missing llms.txt costs you a signpost, not access: the crawlers that matter still reach your pages, still parse them, and still quote them. That is categorically different from a robots.txt line that blocks GPTBot outright, which is rated Critical because it removes you from the pool entirely.

It is not rated Refinement either, and the reason is asymmetry. The work is an afternoon, the risk is nil, and almost nobody has done it, so the expected value is unusually favourable even after discounting the uncertainty heavily. Refinement is for polish on something already working. This is a cheap bet with an unresolved payoff, which is its own category.

Writing one that is actually useful

The failure mode is not a malformed file, it is a useless one: a dump of every URL in your sitemap, which tells a model nothing it could not get from the sitemap. The value is entirely in the editing. You are answering what this site is and where its substance lives, and both answers require judgement a generator cannot supply.

llms.txt

# Example Analytics

> Self-hosted product analytics for small teams. Event
> tracking, funnels and retention, without a data warehouse.

We are a two-person company. The docs below are the
canonical source; the blog is opinion and may be stale.

## Documentation

- [Quickstart](https://example.com/docs/quickstart): install
  and send a first event in about five minutes
- [Event schema](https://example.com/docs/events): the shape
  of every event we accept, with required fields
- [Self-hosting](https://example.com/docs/self-host): Docker
  Compose and Postgres requirements
- [Migrating from Segment](https://example.com/docs/migrate)

## Reference

- [HTTP API](https://example.com/docs/api): endpoints, auth,
  rate limits
- [Pricing](https://example.com/pricing): current limits and
  what they cost

## Optional

- [Changelog](https://example.com/changelog)
- [Engineering blog](https://example.com/blog)

Three things in that file are doing real work. The blockquote is a definition, not a tagline, so a model that reads nothing else can still say what the product is. Each description says what the page contains rather than restating its title. And the note about the blog being stale is the kind of thing only you know, which is exactly the sort of content the file exists to carry.

llms.txt and llms-full.txt are not the same file

/llms.txt is an index: titles, links, one-line descriptions. /llms-full.txt is the content itself, your documentation concatenated into one markdown file so a model can ingest it in a single fetch. The first is a map, the second is the territory.

Most sites should ship the index and think hard before shipping the full dump. A concatenated corpus goes stale silently, can grow past a useful context length, and hands over a tidy copy of everything you have written. Plenty of documentation sites publish both deliberately; plenty of others have good reasons to stop at the index. Our checks report them separately rather than treating the pair as one score, because the second one is a decision rather than an oversight.

Where the file goes on your stack

Next.js: Drop it in public/llms.txt and it is served at the root. If you would rather generate it from the same source as your docs, a route handler at app/llms.txt/route.ts returning text/plain works and keeps the file from drifting out of date.
WordPress: Upload it to the web root next to robots.txt. Do not put it in the theme directory, which is where it usually ends up and where nothing will find it. If a plugin is generating a virtual robots.txt, the same plugin often has somewhere to register another root file.
Shopify: This is the honest limitation. Shopify does not let you write arbitrary files to the store root, so /llms.txt is not straightforwardly available on a stock store. A template rendered at a normal path is reachable but is not at the well-known location, which is the only thing that makes the convention work. If your storefront sits behind a proxy or a custom domain you control, serve it from there instead.
Webflow and other hosted builders: Same constraint as Shopify: root-level static files are generally not yours to add, and a page at /llms-txt is not the same thing. Check whether your plan exposes custom file hosting before spending time on the content.

What an llms.txt does not do

It grants nothing and blocks nothing. It is not a permissions file.

This is the most common misreading, and it runs in both directions. Publishing an llms.txt does not invite crawlers that your robots.txt disallows, and it does not opt you into training. Equally, omitting one does not keep anyone out. Access is decided in robots.txt and by the crawler's own behaviour; llms.txt only describes what is already public.

Which produces the ordering that matters: check that the crawlers are allowed in before writing a file for them to read. A beautifully edited llms.txt on a site that disallows GPTBot is a signpost on a locked door.

Does this affect AI search?

This is the one check where the AI answer is the only answer. llms.txt has no search benefit, no ranking effect and no bearing on how Googlebot understands your site. If it does nothing for assistants, it does nothing at all.

The mechanism it is betting on is context selection. An assistant answering a question about your product has to decide which of your pages to fetch, and today it makes that decision from links, titles and its own index. A curated file that says the quickstart is canonical and the blog is stale is a better input to that decision than a sitemap listing four hundred URLs of equal weight. Whether any vendor is using that input yet is exactly the thing nobody can confirm.

So: write it, keep it short, keep it current, and do not build a strategy on it. The checks that measurably change whether a model can read you at all, whether your content survives without JavaScript and whether the AI crawlers are permitted, are the ones to fix first. The full AI search group covers them.

Confirming it is live

Request https://yoursite.com/llms.txt in a private window and confirm you get plain text with a 200, not a styled 404 and not a download prompt. A file served as text/html still works for most fetchers but is a sign it is being routed through your page renderer rather than served as a static file.

Then check the links in it resolve, because a stale llms.txt is worse than none: it is a curated list actively pointing at pages you moved. Re-crawl and the check flips to found. The one after that, whether what is inside it is any good, is not something a crawler can score for you.

Questions this check raises

Do AI companies actually read llms.txt?: No major provider has committed to it, and there is no public evidence of it being used at scale. It is a proposal from 2024 that has been adopted by a number of documentation sites and by some developer tools. Treat it as cheap and speculative rather than as a requirement.
What is the difference between llms.txt and llms-full.txt?: llms.txt is an index: a short description of the site and a linked list of its important pages, each with a sentence. llms-full.txt is the content itself, every page concatenated into one file. The first is a map, the second is the territory, and they serve different requests.
Is llms.txt the same as robots.txt for AI?: No, and the confusion is worth avoiding. robots.txt controls access and is honoured by real crawlers today. llms.txt controls nothing: it is a courtesy summary. Blocking or allowing an AI crawler is a robots.txt decision, and publishing llms.txt does not change what any bot is permitted to do.