Skip to content
pageinspection

AI bot directives: what your robots.txt actually says about each agent

Ask four people at your company whether AI crawlers are allowed on the site and you will get four answers. Only the file counts, and on most sites the file has never mentioned an AI agent in its life. This check is not about whether your answer is right. It is about whether anyone ever wrote one down.

The short answer

The policy you believe you have and the policy on disk are routinely different, because robots.txt matching is not additive: a crawler that finds a group naming its own token obeys that group alone and ignores the wildcard entirely. A named group missing an Allow line blocks a bot the wildcard would have permitted.

ImportantAudit check · AI bot directives

The policy in your head and the policy on disk

Somebody will ask eventually. A customer with a content licence, a legal review, a partner doing diligence, a journalist. The question is always the same shape: what is your position on AI crawlers. And the answer is not what the marketing lead believes, it is whatever /robots.txt resolves to for each agent at that moment.

There is no third state in that file. An agent with no rule matching it is permitted, so a site that has never considered the question is permitting everything, and the sibling check exists to tell you when the opposite happened by accident. This one asks something narrower and, over a few years, more useful: for each agent you care about, does the file contain a statement, or does it contain a silence you are choosing to interpret.

The difference is not academic. A silence is not attached to a reason, a date, or a person. The next engineer to open the file cannot tell which lines were decided and which were inherited from a template, so they edit around all of them, and the wildcard group grows a rule that quietly changes the position for every agent nobody named.

Allowing an agent and having decided to allow it produce identical HTTP traffic and completely different conversations.

robots.txt and the robots meta tag are answering different questions

These get used interchangeably and they are not interchangeable. robots.txt is about the fetch: which paths an agent may request. The page-level robots meta tag is about what may be done with a page that has already been fetched. One is a door, the other is a set of terms attached to what is inside.

For AI agents specifically, the meta side is thinner than people assume. There is no broadly supported per-agent meta directive: noai and noimageaiwere proposed and picked up by a handful of hosts, and treating them as a control rather than a statement of intent will mislead you. What does exist, and is documented, is Google's snippet family. nosnippet, max-snippet and the data-nosnippet attribute are specified as applying to its AI experiences as well as to ordinary results.

robots.txt cannot express "read this, but do not quote it". A snippet directive can.

That is the gap worth knowing about, because it is the only way to record a per-page exception. robots.txt reasons about path patterns, so a rule for one page means adding a path to a global file. If your requirement is that a particular page may inform an answer without being reproduced in it, the lever is on the page, not in the file.

Which is why the widget below quotes the page's own robots meta alongside the robots.txt verdict when one is present. Two mechanisms, read together, because sites routinely set one while believing they set the other. The page-level directives have their own check for exactly that reason.

What counts as an explicit rule

The parser strips comments, splits each remaining line at its first colon, and treats consecutive User-agent lines as one group sharing the rules beneath them. Then it asks one question per agent: is there a group whose user-agent list contains this name. Three consequences are worth knowing before you read a verdict.

  • Scope is irrelevant. A group for GPTBot containing nothing but Disallow: /internal/ counts as an explicit rule. You named the agent and stated something about it, which is all this check claims to measure. Only the access check cares whether the rule is a root disallow.
  • A group with no rules at all still counts. User-agent: ClaudeBot followed by only a Crawl-delay resolves as named and permitted. That reads like a false positive and is deliberate: the agent appears in the file, so someone considered it.
  • A comment counts for nothing. # we are fine with GPTBot is removed before parsing, exactly as a crawler removes it. Comments are for the humans who maintain the file, and they are load-bearing for that job, but they state nothing to a machine.

One failure mode to recognise in your own result. If /robots.txtis being served by your framework's catch-all route and returns an HTML page with a 200, almost nothing in it parses as a user-agent line, and every agent comes back as not mentioned. The verdict is accurate about the file that was served, which is the same file the crawlers received.

See which AI agents your robots.txt names

Paste any URL on the site. This fetches the real robots.txt and reports how many of the agents we track have a rule written for them, plus the page's own robots meta if it carries one.

See which AI agents your robots.txt names

No signup required. Each free search audits one page, paste any URL to see it in action.

Writing a file that a stranger can audit

The target is not a shorter file. It is a file where someone who has never met you can read the position, see when it was taken, and find the reasoning, without asking anyone.

robots.txt
# AI agent policy: decided 2026-02, reviewed each June.
# Owner: web platform. Full reasoning in docs/ai-policy.md.
#
# House rule: one group per agent, even where the rule is
# identical. Appending a name to an existing group inherits
# whatever is under it, which is how positions get taken by
# accident.

User-agent: GPTBot
Allow: /
Disallow: /internal/

User-agent: ClaudeBot
Allow: /
Disallow: /internal/

# Retrieval with attribution. The one we most want reading us.
User-agent: PerplexityBot
Allow: /
Disallow: /internal/

# Not a fetcher: a control token for Gemini and related
# features. Named so the absence cannot be read as an oversight.
User-agent: Google-Extended
Allow: /

# Declined: bulk corpus, nothing cites back. Revisit if that
# changes, and note the date here when it does.
User-agent: CCBot
Disallow: /

# Anything not named above lands here, so the default is
# written down rather than assumed.
User-agent: *
Allow: /
Disallow: /internal/

Sitemap: https://example.com/sitemap.xml

Four habits are doing the work in that file, and none of them are syntax. Every agent gets its own group, so a future edit cannot inherit a rule it was never meant to. The comments record intent and a date rather than restating the line below them. The wildcard group is stated explicitly instead of left implied. And the reasoning lives in a document the comments point at, because a robots.txt is a terrible place to argue a case and a good place to cite one.

The maintenance rule that follows from this: when a new agent shows up in your logs, add a group for it or write down that you looked at it and chose the default. Both are records. Neither takes five minutes.

Why a recorded decision only rates Important

It cannot be Critical, because nothing is lost while it fails. A site with no AI directives at all is fully readable by every agent, ranks exactly as it did, and serves every visitor. The Critical band is reserved for defects that remove something outright, which is the access check's territory and not this one's.

It sits above Refinement for one reason: the state it reports is unstable. A cosmetic defect stays the size it is. An unnamed agent's status is whatever the wildcard group happens to say after the next edit, made by someone who did not know your position because it was never in the file. That is how the Critical failure gets created, and this check is the one that fires beforehand.

Rate it as maintenance debt on a file that decides whether you exist to an entire class of reader.

Reading the file back after you change it

Do not trust the copy in your repository. Ask the live host what it is serving, and count the agents you meant to name:

terminal
curl -s https://example.com/robots.txt | grep -in '^user-agent'

# Expect one line per agent, and no two of them adjacent.
# Two consecutive User-agent lines are one group, which is
# almost never what the person who added the second meant.

Then make it hard to lose. A robots.txt is the file most likely to be replaced by a platform without anyone noticing: an SEO plugin generating a virtual one, a proxy rule, a migration that reinstates the framework default. A single assertion in CI that the deployed file contains each agent name turns that from a discovery you make in eight months into a failed build.

Re-run the widget above and the verdict should move to a pass, with every agent named. Keep the reasoning document current in the same commit as the file, because the value of the whole exercise is that the next person does not have to reconstruct your thinking. If you are about to publish a full-content file as well, check that the rules you just wrote do not disallow the path it lives at.

Questions this check raises

Does a named user-agent group inherit rules from the wildcard group?
No, and this is the mistake that produces most accidental blocks. A crawler uses the single most specific group matching its token and ignores every other group, including the wildcard. Adding a group for a bot means restating everything you want that bot to be allowed to do.
Is the robots meta tag the same as robots.txt for AI bots?
No, they answer different questions. robots.txt decides whether the bot may fetch the page at all; the meta robots tag decides what may be done with it once fetched. A page blocked in robots.txt is never fetched, so its meta tag is never read.
How do I verify my robots.txt says what I think?
Read it back agent by agent rather than as a whole. For each bot you care about, find the group that would match it, and check that group in isolation as though the rest of the file did not exist. That is what the crawler does, and it is the only reading that predicts behaviour.