Get read by AI. Then get cited by it
Assistants send traffic to pages they can reach, parse and quote, in that order. Most sites fail at the first step and never find out, because no classic SEO tool looks.
These checks target the surfaces people now ask instead of searching: ChatGPT, Perplexity, Claude, Gemini, Google AI Overviews.
Your next visitor may never see a results page
People increasingly ask instead of search. The answer arrives with two or three citations attached, not ten blue links, and ranking fourth is worth nothing when the assistant quotes three sources. Whether you are one of them is decided by things classic SEO tools were never built to check.
Part one: can an assistant read your page?
Access and readability come before everything else. A model that cannot reach the page has nothing to choose from, and no amount of good writing changes that.
Can AI crawlers reach you?
GPTBot, ClaudeBot, PerplexityBot and Google-Extended each need explicit permission. Plenty of sites block them in robots.txt without ever deciding to, often inherited from a template or a CDN default.
Have you told them what you are?
llms.txt is robots.txt for language models: a short, plain file describing your site and pointing at the content that matters. Almost nobody has one yet, which makes it one of the few places you can still be early.
Can they read it without a browser?
Most AI crawlers don't execute JavaScript. If your copy only appears after hydration, the model sees an empty shell, and you are invisible to it no matter how good the content is.
Can they quote you cleanly?
Models lift self-contained passages. Semantic headings, real lists, and an answer stated before the preamble decide whether your page gets quoted or skimmed past.
Part two: will it quote you, or someone else?
Readability gets you into the candidate pool. Packaging decides whether you are the one quoted. AEO is rarely about writing more. It is about making what you already published liftable.
Questions you answer but never claimed
We find pages that clearly answer a question in their headings yet carry no FAQ markup. It is the cheapest ranking win most sites are sitting on.
Answers buried under warm-up
Answer engines lift the first passage that stands on its own. If yours arrives in paragraph six, after the history of the topic, it never gets read.
Prose where a table would win
Snippets favour lists, tables and definitions. We flag content whose shape is fighting the format the engine wants to display.
Nothing marked safe to read aloud
Speakable markup tells a voice assistant which sentences it can say verbatim. Without it, assistants paraphrase someone else instead.
How to fix each one
Five changes, in the order worth doing them. Every sample below is meant to be copied and used as it stands.
Allow the AI crawlers
Each assistant uses its own user agent, and a rule for one says nothing about the others. Vendors also split crawling from retrieval: GPTBot crawls broadly, while OAI-SearchBot fetches a page because someone asked about it. Google-Extended is the odd one out and worth understanding before you touch it, because it is not a crawler at all.
Confirm it worked: Fetch https://yourdomain.com/robots.txt in a browser and read it top to bottom. A bare Disallow: / under User-agent: * blocks these agents too unless you name them explicitly.
# Assistants that cite their sources. Allowing them is what makes
# your pages eligible to be quoted back to a reader.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Not a crawler. This token only controls whether content Google has
# already fetched may be used for Gemini grounding and model training.
# It has no effect on Google Search indexing or ranking.
User-agent: Google-Extended
Allow: /
Sitemap: https://example.com/sitemap.xmlWrite an llms.txt
A markdown file at the root: an H1 with your name, a blockquote summarising the site in a sentence, then H2 sections listing the pages that matter as markdown links with short notes. An Optional section marks what can be skipped when context is tight. Keep it short. It describes the site, it is not a copy of it.
Confirm it worked: Serve it at /llms.txt as text/plain or text/markdown and open it directly. Be realistic about impact: this is a community proposal, and no major vendor has publicly committed to consuming it yet.
# Example Co
> Short description of what this site is and who it serves, in one sentence.
An optional paragraph for context a model could not infer from page titles
alone: what you do, what you do not do, and which terms you use for things.
## Docs
- [Getting started](https://example.com/docs/start): install and first run
- [API reference](https://example.com/docs/api): every endpoint, with examples
## Product
- [Pricing](https://example.com/pricing): plans, limits and what is included
## Optional
- [Changelog](https://example.com/changelog): release historyMake the content survive without JavaScript
Most AI crawlers read the HTML your server returns and never run your bundle. If the page ships an empty shell and paints the content on hydration, the model sees the shell. Server-rendering or prerendering the content that matters fixes it; you do not need to abandon a client framework to do it.
Confirm it worked: Run the command below against a real page, or open DevTools, use the command palette to Disable JavaScript, and reload. What remains is roughly what a model gets.
# Fetch the page the way a crawler without a browser does, and look
# for a sentence that only exists in your rendered body copy.
curl -sL https://example.com/your-page | grep -c "a sentence from your page"
# 0 the text is not in the HTML. It is drawn by JavaScript, and most
# AI crawlers will never see it.
# 1+ the text ships in the HTML. A model reading the raw response finds it.Restructure to answer-first
The passage a model lifts is usually the first self-contained one under a relevant heading. Put the answer there, in a sentence or two that makes sense with no surrounding context, and move the history, the caveats and the background underneath it. The page reads better for humans in a hurry too.
Confirm it worked: Read the first two sentences under each heading on their own. If they do not answer the heading, the extractable passage is somewhere else, or nowhere.
<!-- Before: the answer arrives after the throat-clearing -->
## Does llms.txt affect Google rankings?
Robots exclusion has a long history, going back to 1994 when...
Standards bodies have debated machine-readable site descriptions for...
Opinions among practitioners differ, with some arguing that...
No, llms.txt has no effect on Google Search rankings.
<!-- After: the answer is the first thing under the heading -->
## Does llms.txt affect Google rankings?
No. llms.txt has no effect on Google Search rankings. It is a separate,
emerging convention aimed at language models, not at search crawlers.
Background: robots exclusion dates to 1994, and llms.txt borrows the idea
of a small file at a predictable path...Add FAQ and speakable markup
FAQPage markup tells a parser which text answers which question, rather than making it infer that from your layout. Only use it where the page genuinely asks and answers questions. Speakable marks the sentences safe to read aloud, though Google supports it narrowly, for news content in a limited set of locales, so treat it as an option rather than a general win.
Confirm it worked: Paste the rendered page into Google's Rich Results Test. The markup must match the visible text: schema that disagrees with the page is a manual-action risk, not a shortcut.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "Does llms.txt affect Google rankings?",
"acceptedAnswer": {
"@type": "Answer",
"text": "No. llms.txt is a separate convention aimed at language
models and has no effect on Google Search rankings."
}
}
]
}
</script><script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "WebPage",
"name": "Does llms.txt affect Google rankings?",
"speakable": {
"@type": "SpeakableSpecification",
"cssSelector": [".answer-summary"]
}
}
</script>The 16 checks behind this
What the audit inspects on every page. Full descriptions live on the catalog rather than being repeated here.
AI search (GEO)
9 checks- llms.txtImportant
- llms-full.txtRefinement
- AI crawler accessibilityCritical
- AI bot directivesImportant
- Text extractabilityCritical
- Content segmentationImportant
- Answer-first contentImportant
- Question markupImportant
- AI-friendly structureImportant
Answer engines (AEO)
7 checks- FAQ opportunitiesImportant
- HowTo opportunitiesImportant
- Direct answer sectionsImportant
- Question-based headingsRefinement
- Speakable contentRefinement
- Structured answersImportant
- Featured snippet readinessCritical
Why this is not a manual job
Checking these 16 items by hand means fetching robots.txt, testing four user agents against it, disabling JavaScript on a sample of pages, reading the schema on each one, and judging answer placement page by page. That is an afternoon for one page and a week for a site.
One crawl does it across every page and exports each failure as a fix you can run.
See what a fix brief containsStraight answers on a young topic
Where the honest answer is that nobody knows yet, that is what it says.
Write one, but keep your expectations honest. It is a community proposal rather than a ratified standard, and no major assistant vendor has publicly committed to consuming it, so nobody can promise you traffic from it today. It also costs an afternoon at most, it cannot hurt anything, and being early is occasionally worth something. Treat it as a cheap option, not a growth lever.
Both, and you have to decide which matters more for your site. Blocking does reduce the chance of your text being absorbed into a model, but it also removes you from the pool of pages an assistant can cite, which is where a growing share of discovery now happens. Publishers with licensing leverage sometimes block deliberately. If your pages exist to be found, blocking is usually paying a real cost to avoid a theoretical one.
Not exactly, and the distinction matters. Vendors run different agents for different purposes: some crawl broadly, some fetch a single page because a user asked about it right now, and the rules can be set separately per agent in robots.txt. Google-Extended is the clearest example, since it is not a crawler at all, only a control over whether content Google already has can feed Gemini. If your concern is training specifically, allow the retrieval agents and disallow the training ones rather than blocking everything.
It is early and the honest answer is that nobody has durable numbers yet. What can be said without hand-waving: the volume is smaller than classic search, the intent tends to be further along because the assistant has already done the filtering, and a citation is a recommendation rather than a blue link. We are not going to quote you a conversion rate we cannot source.
Ranking well means Google can reach and understand you, which is most of the work and puts you ahead of most sites. It does not mean your answer is packaged to be lifted. Snippets, answer boxes and assistants all pick the passage that stands on its own, so a page that ranks first and buries its answer in paragraph six regularly loses the quote to a page ranked below it.
Traditional crawlers are excellent at the things search engines have cared about for twenty years: status codes, canonicals, titles, redirects, hreflang. What they generally do not check is whether GPTBot, ClaudeBot or PerplexityBot are allowed at all, whether an llms.txt exists, whether your content survives with JavaScript disabled, or whether a passage is structured tightly enough to be quoted. Those are the 16 checks on this page.
Find out if AI can read your site
One crawl tells you whether the assistants your customers use can see you at all.
No signup required. Each free search audits one page, paste any URL to see it in action.