llms-full.txt: handing over your whole corpus in one file
Twenty lines of build script will concatenate your documentation into one file this afternoon. Whether that file should be served at /llms-full.txt is a separate question, and for most sites the answer is no. This is the one check in the AI group where passing is not automatically better than failing.
The short answer
llms-full.txt concatenates your site content into one file so a model can read it in a single fetch. Generating it is trivial and deciding to publish is not: it is a second copy of everything you publish, it goes stale independently, and past a few hundred thousand tokens no client will read it whole anyway.
Generating it is trivial, deciding to is not
The index file describes your content: titles, links, a line each. This one is the content, every page concatenated into a single plain-text document. One is a map with your judgement written on it. The other is the territory, in bulk, with the navigation removed.
That last clause is the decision. A reader who fetches your corpus never visits a page. There is no nav, no related links, no sign-up path, no analytics event, and no next step you designed. For a documentation site that is precisely the point: the job is to be consumable, and a developer who gets an accurate answer is a developer who stays. For a site whose pages carry a conversion path, you have handed over the text and thrown away the page.
Publishing one is a business decision wearing a build-step costume.
It also fails differently from the index. A stale llms.txt points at pages that 404, which is visible to anyone who checks. A stale corpus serves last quarter's answers, in your voice, with no indication that anything is out of date. Nobody complains, because nobody can tell.
Who fetches this file today
The honest inventory, because the marketing version of this section is invented. No assistant vendor has publicly committed to fetching /llms-full.txt, there is no registry of consumers, and any citation or traffic figure you are quoted for it is made up.
What does demonstrably happen is people. An engineer pastes the URL into a chat window to ask a question about your API. A coding agent is told to read the docs for a library before writing against it. Somebody building an internal assistant wants your reference material without maintaining a scraper. All three are deliberate, human-initiated fetches rather than a pipeline you are hoping exists.
Which tells you exactly which sites this is for. If your content is reference material somebody would want in full, an API surface, a configuration schema, a component library, a standards document, the file has a real audience today. If your content is a blog, a catalogue, a marketing site or a set of individually complete articles, nobody wants all of it at once, and you are generating a file to pass a check.
The size ceiling nobody warns you about
Whoever reads this file has a context limit, and your file has no say in how that limit is enforced. As a back-of-envelope figure, four characters is roughly one token, so a two megabyte corpus is asking for something in the region of half a million tokens. The fetch either fails, or gets truncated, and truncation is the outcome to think about because it is silent.
The cut lands where the consumer decides, which is almost never a section boundary. Your first few chapters arrive intact and the last third of the file is simply absent, and nothing anywhere reports that. The file appears to be working. It is working for the parts of your documentation that happen to sort early.
Two mitigations, both structural. Scope the file: one per product area, listed in the index, rather than one enormous file per site. And order it deliberately, most valuable reference first, since a truncating reader always keeps the beginning. Then say the size out loud in the file itself so a reader can decide before committing a context window to it.
# Example Analytics documentation
> Full text of the docs at https://example.com/docs.
> Generated 2026-02-14 from content/docs. 34 pages, 412 KB.
> The index at https://example.com/llms.txt is smaller and
> may be the file you want.
# Quickstart
Source: https://example.com/docs/quickstart
Install the SDK, set your write key, and send a first event.
The whole path takes about five minutes...
---
# Event schema
Source: https://example.com/docs/events
Every event we accept carries four required fields...The Source: line above each section is the detail most generators skip and the one that decides whether this file ever sends you a visitor. Without it, a model quoting your corpus has a paragraph and no URL, so the best you can hope for is an uncredited answer.
Check a site for llms-full.txt
Paste any URL on the site. The result is site-level: whether /llms-full.txt returns a document, and how that reads next to the index. A pass here is recorded as a choice rather than a win.
No signup required. Each free search audits one page, paste any URL to see it in action.
Why this one sits at the bottom of the scale
The severity scale rates what the failure costs you. Here the failure state is having no llms-full.txt, which is the correct configuration for most of the web. Rating it any higher would mean the audit telling the majority of the sites it looks at to do something they should not do, which is worse than useless: it is a report that has to be argued with.
Compare it with its own sibling. A missing index is rated higher because absence there has no upside at all: it costs an afternoon, it risks nothing, and there is no site for which not having one is the better answer. Every part of that reasoning inverts for the corpus. The cost is a maintained build artifact, the risk is a silently stale one, and for plenty of sites the better answer is genuinely not to ship it.
So Refinement is not a shrug here, it is the accurate reading. The check exists because the pair should be reported separately rather than scored as one AI-readiness number, and because a site that has published one deliberately deserves to see that recorded.
Building it from the same source as your docs
There is one rule and everything else follows from it: never write this file by hand and never commit an edited copy. It must be generated in the build, from the same source your pages render from, or the two will diverge and you will not find out.
// Wired into the build, not run by hand:
// "prebuild": "node scripts/llms-full.mjs"
import { readdir, readFile, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
const DIR = 'content/docs'
const SITE = 'https://example.com'
const STAMP = new Date().toISOString().slice(0, 10)
const files = (await readdir(DIR)).filter((f) => f.endsWith('.mdx')).sort()
const sections = []
for (const file of files) {
const raw = await readFile(join(DIR, file), 'utf8')
const parts = raw.split(/^---\s*$/m)
const meta = parts.length > 2 ? parts[1] : ''
const text = (parts.length > 2 ? parts.slice(2).join('---') : raw).trim()
const title = (/^title:\s*(.+)$/m.exec(meta) || [, file])[1].trim()
const url = SITE + '/docs/' + file.replace(/\.mdx$/, '')
// The source URL is what lets a model cite the page
// rather than the dump.
sections.push(['# ' + title, 'Source: ' + url, '', text].join('\n'))
}
const corpus = sections.join('\n\n---\n\n')
const kb = Math.round(corpus.length / 1024)
const header = [
'# Example Analytics documentation',
'',
'> Full text of the docs at ' + SITE + '/docs.',
'> Generated ' + STAMP + ' from ' + DIR + '. ' +
files.length + ' pages, ' + kb + ' KB.',
'> The index at ' + SITE + '/llms.txt is smaller and',
'> may be the file you want.',
'',
].join('\n')
await writeFile('public/llms-full.txt', header + corpus + '\n')
console.log('llms-full.txt:', files.length, 'pages,', kb, 'KB')Three properties of that script are what make it safe to leave running for two years. The stamp and the page count are written into the file, so staleness is readable rather than inferred. The source URL sits above every section. And the whole thing is derived, which means the review question at the end of a docs pull request is never "did anyone update the corpus".
If you would rather not commit a generated artifact, serve it from a route instead. In Next.js a handler at app/llms-full.txt/route.ts returning text/plain does the same job at request time, which trades a build step for a cache policy you now have to think about.
A second copy of every page you already publish
By construction this file duplicates your entire site's text at one URL. That is not the classic duplicate-content problem, since nothing is competing for a ranking here and a plain-text file has nowhere to put a canonical tag. But if you would rather it stayed out of search results entirely, the meta tag route does not exist for you and the response header does:
Content-Type: text/plain; charset=utf-8
X-Robots-Tag: noindex
Cache-Control: public, max-age=3600
# noindex, not Disallow. A robots.txt block stops the
# agents you published the file for from reading it,
# which is the one contradiction to avoid here.That contradiction is common enough to check for deliberately. Publishing your whole corpus for AI agents while robots.txt disallows those agents is a decision nobody would defend out loud, and it happens because the two changes were made months apart by different people.
The other consequence is worth naming plainly rather than dramatising. This file lets anyone take your complete documentation in a single request. That was already possible with your sitemap and a loop; what you are removing is the friction. Removing friction for the reader you want and the scraper you do not is the same action, and there is no version of this file that does one without the other.
Does this affect AI search?
Not measurably, today. No vendor consumes it on a published commitment, so there is no mechanism by which shipping one improves how often you are cited, and nobody honest can tell you otherwise.
What it changes is the cost of reading you for a consumer who has already decided to. One fetch instead of forty, plain text instead of rendered HTML, no JavaScript step, no rate limit. That matters most where an answer genuinely spans pages, a configuration reference where the correct response depends on three sections at once, and matters not at all where any single page fully answers the question. If your pages are self-contained, an assistant that reads one of them has everything it needed.
For search it does nothing whatsoever. A text file at a well-known path is not a ranking surface, and treating this as an SEO task will waste a sprint. The checks in the AI search group that decide whether a model can read you at all come first, every time.
If you are choosing between the index and the corpus, ship the index.
It is smaller, it carries your judgement about what matters, it cannot go stale in a way nobody notices, and it is the file the convention was actually built around. The corpus is the follow-up for sites whose content people want in bulk, not the next rung of a ladder everyone should climb.
Confirming what you shipped is current
The existence test is the easy half. Request the file, confirm plain text and a 200, and check the byte count against the figure the header claims:
curl -s https://example.com/llms-full.txt | head -8
curl -s https://example.com/llms-full.txt | wc -c
# Then the test that actually catches a stale build: grep
# for a sentence you added to the docs today.
curl -s https://example.com/llms-full.txt | grep -c 'a phrase you just wrote'That last command is the one worth keeping. A generated file behind a CDN or a cached build step will keep serving a correct-looking corpus long after the docs moved on, and the date in the header is only proof of when the script ran, not of what it read. If the grep returns zero, your pipeline is fine and your cache is not.
Two more things to rule out. If the file lives in public/ and the script writes there, make sure an early hand-written copy is not also committed, because whichever wins is decided by build order rather than by you. And re-run the check above: it should report the file as found, and the sibling index alongside it, which is the pairing you want the report to show.
Questions this check raises
- How large can llms-full.txt be?
- There is no specified limit, and the practical ceiling is whatever a client is willing to fetch and fit in a context window. Past a few hundred thousand tokens the file stops being useful as a single read, and a smaller llms.txt index pointing at individual pages serves the same purpose better.
- Should I publish llms-full.txt?
- Only if you can generate it from the same source as your pages, so it cannot go stale. A hand-maintained copy of your whole site is a second version of everything, and a stale one is worse than none: it hands a model confident, wrong information with your domain attached to it.
- Does llms-full.txt help with AI search?
- There is no measured effect, and anyone claiming one is guessing. The defensible reason to publish is that it costs nothing when generated and gives a client that wants your content in one fetch a way to get it. That is a small, real benefit rather than a ranking mechanism.