Are GPTBot and ClaudeBot allowed on your site? How to check and decide
The sites most thoroughly excluded from AI answers are rarely the ones that decided to be. They are the ones that pasted a robots.txt block from a forum thread in 2023, or added a bot name above an existing rule without reading what it inherited. This is the check that tells you which you are.
The short answer
Most sites that block AI crawlers did not decide to. It arrives as a security plugin default, a CDN bot-management rule, or a copied robots.txt from a site with a different policy. Not being mentioned in robots.txt means allowed, so the accident is always an explicit rule someone added without reading what it covered.
The accident this check exists to catch
robots.txt has one syntax feature that causes almost all of the damage: consecutive User-agent lines share the rules beneath them. So a file that already contains a blocked group, a staging crawler, a scraper someone got annoyed by, will silently apply those rules to any agent name appended above them.
# What someone thought they were adding:
# "just note GPTBot here, we will decide later"
User-agent: SomeScraper
User-agent: GPTBot
Disallow: /
# Both agents are in ONE group. GPTBot is now
# disallowed from the entire site. Nothing warns you,
# nothing logs it, and the file still looks tidy.The other route in is inheritance from the wildcard. A site with User-agent: * followed by Disallow: /, left over from a pre-launch block that nobody removed, blocks every AI crawler along with everything else. That one at least blocks search engines too, so it usually gets noticed. The group-stacking version can sit there for years.
Which agents matter, and what each one actually does
Blocking "AI bots" as a category is where most decisions go wrong, because the agents do different jobs and only some of them are what people mean to object to.
- GPTBot
- OpenAI's bulk crawler, collecting content that may be used to train future models. This is the one with the clearest case for blocking if your objection is to training, and blocking it does not remove you from ChatGPT's ability to fetch a page a user asked about.
- ClaudeBot
- Anthropic's crawler. Anthropic publishes separate agent names for different purposes, which means the fine-grained decision is available to you rather than all-or-nothing.
- PerplexityBot
- Retrieval and citation oriented: Perplexity's product is answers with sources attached, so this agent is the one most directly connected to being cited and linked. Blocking it removes you from a surface that sends actual clicks.
- Google-Extended
- The special case, and the one most often misunderstood: it is not a crawler and never fetches anything. It is a control token that says whether content Googlebot already has may be used in Gemini and related features. Disallowing it does not reduce your crawling or your search ranking, and allowing it does not add a single request to your server.
- CCBot
- Common Crawl, a public dataset rather than one company's model. Many models have been trained on it, so blocking it is broad and indirect: you are removing yourself from a corpus rather than from a product.
What we parse out of robots.txt
We fetch /robots.txt, parse it into groups the way the specification says to ( stacked user-agent lines sharing one rule set), and resolve each named AI agent to one of three states: blocked by a root disallow, named and permitted, or not mentioned at all.
The resolution order matters and mirrors how a well-behaved crawler decides. An agent's own group wins if it has one. Failing that, the wildcard group applies. Failing that, the default is permission. Note the asymmetry inside a group: an Allow: / alongside a Disallow: / resolves as permitted, which is how sites carve one agent out of a broad block.
Check a site for blocked AI crawlers
Paste any URL on the site. This fetches the site's real robots.txt and reports where each of the five agents stands. It is the most directly useful of these widgets, because the answer is usually a surprise.
No signup required. Each free search audits one page, paste any URL to see it in action.
Why not mentioned means yes
This is the part that changes people's minds about their own file. robots.txt has no concept of "undecided". An agent with no rule matching it is permitted, because the protocol is an opt-out and always has been.
Which means a site that has never thought about AI crawlers is allowing all of them, and a site that blocked two of them has made a decision about those two and an accidental decision about the rest. Neither state is neutral. If you want a considered position, every agent you care about has to be named.
There is no way to abstain. Silence is consent, in this protocol by design.
Separating retrieval from training
This is the decision worth actually making, and it is available to you: block the agents that collect for training, permit the ones that fetch because a user asked a question right now. You keep the citations and the clicks, and you decline the corpus.
# Retrieval agents: permitted. These fetch a page
# because someone asked about it, and cite it.
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# Bulk training crawlers: declined.
# Each in its OWN group, so no rule is inherited by
# an agent it was not written for.
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
# Not a crawler. Controls whether content Google
# already holds may feed Gemini.
User-agent: Google-Extended
Disallow: /
# Search unaffected. Spelled out so nobody
# "tidies up" the file and takes this with it.
User-agent: Googlebot
Allow: /Two habits make this file survive contact with future editors. Give every agent its own group even when several share the same rule, so appending a name can never inherit something unintended. And comment the intent rather than the mechanism, because the next person to edit it will be guessing at why, not at what.
What blocking gains you, and what it costs in AI search
The honest version of both sides, since this is a real decision and not a checkbox.
What blocking genuinely gets you. Less of your text absorbed into training corpora. For a publisher with licensing leverage, a negotiating position: you cannot sell access you have already given away. For a site whose content is the product, a defensible reduction in wholesale copying.
What it costs. Removal from the pool of pages an assistant can cite, on a surface where a growing share of research now starts. A citation is closer to a recommendation than a blue link, and the traffic that does arrive tends to be further along because the assistant has already filtered. Block retrieval and you are not in that conversation, whatever you rank for.
The asymmetry that decides it for most sites. If your pages exist to be found, blocking pays a certain, immediate cost to avoid a diffuse, uncertain one. If your pages are the thing you sell, the calculation genuinely inverts. What is not defensible is arriving at either position by accident, which is what the great majority of blocked sites have done.
robots.txt is a request, not a fence.
Well-behaved crawlers honour it and it has no enforcement mechanism whatsoever. If your requirement is that content cannot be taken, robots.txt is the wrong tool and authentication is the right one. What this file does is record a preference that reputable operators respect.
Why this is Critical
It is the only check in the AI group whose failure is total. Everything else on that list, llms.txt, answer structure, question markup, changes how well a model can use your page. A root disallow changes whether there is a page at all, from that agent's point of view.
It is also site-wide by nature and invisible without looking. One line in one file, no error, no report, nothing in analytics, applied to every URL you have. Critical is for defects that are total, silent and sitewide, and this is all three.
Confirming the file says what you meant
Read the whole file top to bottom, not the group you just edited, because the failure mode here is inheritance from something above your change. Specifically: check whether your new User-agent line is directly above another User-agent line, and if it is, whose rules it just joined.
Then confirm from outside. Fetch https://yoursite.com/robots.txt and check you get plain text rather than a rendered page, since a framework catch-all route can shadow the static file and serve HTML that no parser will read. Re-crawl and the report should name each agent and its state, which is the version worth keeping a screenshot of before your next migration.
One last thing worth doing while you are here: access is necessary and not sufficient. A permitted crawler still gets nothing useful from a page whose content requires JavaScript, which is the next check in the group, and it gets a signpost from an llms.txt only once it is allowed through the door.
Questions this check raises
- Does blocking GPTBot remove me from ChatGPT?
- Not from ChatGPT search, which uses a different agent. GPTBot collects training data, OAI-SearchBot builds the search index, and ChatGPT-User fetches a page live because someone asked about it. Blocking the first opts you out of training and leaves the other two, and blocking all three because they share a prefix removes you from a surface you probably wanted.
- If robots.txt does not mention an AI bot, is it blocked?
- No. Not mentioned means allowed, in every mainstream implementation. Blocking requires an explicit rule, which is why an accidental block is always something someone added rather than something they forgot.
- Should I block AI crawlers?
- It depends on whether your content is the product. A publisher whose archive is the asset has a real argument for blocking training crawlers. A business whose site exists to be found has almost none, and blocking retrieval agents specifically means an assistant asked about you has nothing to cite.