Why can I see the page in my browser but a crawler cannot?

Because your browser arrives with cookies, a logged-in session, a residential IP address and a location the crawler does not have. A page behind a consent wall, a geo-redirect or a soft paywall looks completely normal to you and returns a challenge or a redirect to a bot. Fetching the URL with a plain request and no cookies is the only honest test.

Is crawlable the same as indexable?

No, they are separate states and a page can fail either one independently. Crawlable means a bot can fetch the page; indexable means it is allowed to be stored and served. A page blocked in robots.txt is not crawlable but can still appear in results as a bare URL, and a page returning noindex is crawlable but will not be kept.

How do I test what a bot actually sees?

Request the URL the way a bot does: no cookies, no JavaScript, a bot user agent, and from outside your own network. A plain command-line fetch answers most of it in one line. Search Console URL inspection answers the rest for Google specifically, including whether the render differs from the raw response.

Whether a bot is allowed to fetch the page at all

Every other check in this audit assumes a bot got the page. This is the one that asks whether it did. Most of the things that say no are invisible from a browser, because a browser is not who they are saying no to.

The short answer

Crawlability is whether a bot can complete the walk from your entry point to a page without hitting a wall: robots.txt rules, login gates, geo-redirects, aggressive rate limiting, JavaScript-only navigation, WAF challenges, or infinite parameter traps. Your browser is the wrong instrument for testing it, because your browser has cookies, a session, and a location the bot does not.

CriticalAudit check · Crawlability

Seven walls, and how many of them are yours

Some of these you configured. Some were configured by whoever set up your hosting, and some are a product you bought behaving as advertised.

A robots.txt disallow. The explicit one, and the one people check. Worth remembering that it stops the fetch rather than the indexing, which is why combining it with a noindex produces the worst of both.
A nofollow on the only inbound link. Rare and unpleasant to diagnose, because the page is permitted and simply never nominated. If every route to a page runs through links marked nofollow, the page is effectively unreachable.
Authentication. Anything behind a login is uncrawlable by design, and the mistake is not the wall itself but forgetting which pages moved behind it. A section that used to be public and now requires an account will quietly leave the index.
Bot protection and firewall rules. The most common cause on modern sites and almost never a deliberate decision. A managed rule set that challenges unfamiliar user agents will serve a challenge page to a crawler and your real page to you.
Rate limiting. A crawler requesting fifty pages a second looks exactly like something you installed a rate limiter to stop. The result is a burst of 429s, then a much slower crawl, then sections that are visited rarely enough to go stale.
Geographic and IP restrictions. A rule serving different content, or nothing, outside your own market. Crawlers arrive from where they arrive from, and a country block is a crawl block.
Server errors under load. Not a wall so much as a door that sticks. Sustained 500s teach a crawler to come less often, and the reduced frequency outlasts the incident that caused it.

Notice that four of the seven live in infrastructure rather than in your codebase. That is the single most useful thing to know about this check: the fix is frequently a conversation with whoever owns the CDN, not a change to a template.

Why your browser is the wrong instrument

You are logged in, you have cookies, you are coming from a familiar address, you accept JavaScript, and you have a user agent nobody blocks. A crawler has none of those. Every rule that distinguishes visitors will distinguish you from it, which means the page you are looking at is not evidence about the page it gets.

The specific traps this creates:

A challenge page returns a 200. Bot protection usually answers with a real HTTP success carrying an interstitial. So a status check passes, the content is a challenge, and every content check downstream is measuring the challenge.
The block is intermittent. Reputation-based rules escalate under load and relax afterwards, so a test at eleven in the morning proves nothing about the crawl at three.
Your session is the thing making it work. Pages that render for a logged-in visitor and redirect anonymous ones are common in documentation and in dashboards, and the redirect target is often a login page returning a 200.

Test with the user agent you care about, from outside your network.

Two changes to any check you run: send a crawler user agent, and run it from somewhere that is not your office or your VPN. A great many crawl blocks are invisible from inside a company because the office address was allowlisted years ago and nobody remembers.

The authoritative version of this is the fetch tool in your search console, which requests the page as the crawler itself rather than as something claiming to be it. Anything you run yourself can be given a different answer.

The question is whether the walk arrives

A single fetch tells you what happens when a URL is requested directly. This check asks something different: starting at your home page and following links, does anything ever get here. That is a traversal, not a request.

This one needs the full crawl

The question is whether a bot walking from your home page arrives here, and answering it means attempting the walk rather than fetching a URL somebody already has.

The instant search on this site audits a single page, so rather than show you a verdict it cannot support, this guide sends you to the place the check actually runs.

Run the full audit, free

Crawlable and indexable are separate states

Four combinations, and three of them are things people ship by accident.

states.txt

crawlable  indexable  what you have
---------  ---------  ----------------------------------------------
yes        yes        the normal state
yes        no         a noindex the crawler can read and obey.
                      correct for a filter, a print view, a thank-you
no         yes        blocked in robots.txt but linked from elsewhere.
                      the URL can still appear, with no description,
                      because nothing was allowed to read the page
no         no         invisible, and you cannot tell which of the two
                      reasons is doing it

The third row is the one worth internalising. A robots.txt disallow is not a way to keep a page out of results, because it prevents reading rather than listing. If you want a page gone, it has to be crawlable enough for the noindex to be seen, which is the argument the indexability guide makes at length.

The fourth row is the diagnostic problem. Once both are in place you have removed your own ability to reason about the page, because a report can only tell you it never got in. Fix the crawl block first, then read what the page actually says about itself.

Does this affect AI search?

Yes, and the answer differs from the search one in a way that catches people out. Search crawling and AI crawling are done by different agents with different names, and your robots.txt almost certainly treats them differently, sometimes without anyone having decided to.

The infrastructure walls are worse here. Bot protection products increasingly ship with rules that block AI crawlers by default, or offer a one-click setting that somebody enabled during a different conversation. So a site can be perfectly crawlable for search and closed to everything else, and nothing in a normal audit would mention it.

Which makes this check two checks in practice. Whether a search crawler arrives, and whether the agents that build training and retrieval corpora do. The AI access guide covers the second, including which agents are worth naming and what each one is actually for, and it is worth reading straight after this one because the answers usually differ.

Why permission failures sit at the top of the scale

Because nothing downstream survives them. A page a crawler cannot fetch has no title, no markup, no content and no links, as far as anything is concerned. Every other finding in this audit is about a page that exists being less good than it could be. Critical is reserved for a page not being in the running, and this is the most complete version of that.

The rating is also about scope. Crawl blocks are rarely per page. A robots.txt line, a firewall rule or a rate limit applies to a pattern, a path or a whole host, so the typical finding is a section rather than a URL. One rule can remove a thousand pages, and the pages themselves will look fine to everyone who checks them in a browser.

What makes it urgent rather than merely serious is the delay. A crawl block does not have an immediate effect: pages stay indexed on their existing information for a while, then go stale, then drop. So by the time traffic moves, the change that caused it is weeks or months old and nobody connects the two. That lag is the argument for checking this on a schedule rather than when something looks wrong.

Asking your own site as a bot

Three requests, from outside your network, and the differences between them are the finding.

as-a-bot.sh

URL=https://example.com/guides/carbon-steel

# as you
curl -s -o /dev/null -w 'browser-ish: %{http_code} %{size_download} bytes\n' \
  -A 'Mozilla/5.0' "$URL"

# as a search crawler
curl -s -o /dev/null -w 'googlebot : %{http_code} %{size_download} bytes\n' \
  -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' "$URL"

# with nothing at all, which is what many scripts and agents send
curl -s -o /dev/null -w 'no agent  : %{http_code} %{size_download} bytes\n' \
  -A '' "$URL"

# and is the path disallowed anyway?
curl -s https://example.com/robots.txt | grep -iE '^(user-agent|disallow|allow)'

Compare the byte counts, not just the codes. Three 200s of wildly different sizes means one of those requests received a challenge or a stripped-down page, and the small one is what is being indexed.

If the crawler request is refused or shortened, the next step is your CDN or firewall logs rather than your application. Search for the request by user agent and you will usually find the rule that matched it, with a name somebody chose during a security review and nobody has revisited.

Questions this check raises

Why can I see the page in my browser but a crawler cannot?: Because your browser arrives with cookies, a logged-in session, a residential IP address and a location the crawler does not have. A page behind a consent wall, a geo-redirect or a soft paywall looks completely normal to you and returns a challenge or a redirect to a bot. Fetching the URL with a plain request and no cookies is the only honest test.
Is crawlable the same as indexable?: No, they are separate states and a page can fail either one independently. Crawlable means a bot can fetch the page; indexable means it is allowed to be stored and served. A page blocked in robots.txt is not crawlable but can still appear in results as a bare URL, and a page returning noindex is crawlable but will not be kept.
How do I test what a bot actually sees?: Request the URL the way a bot does: no cookies, no JavaScript, a bot user agent, and from outside your own network. A plain command-line fetch answers most of it in one line. Search Console URL inspection answers the rest for Google specifically, including whether the render differs from the raw response.