What is a crawler trap?

A pattern that generates an unbounded number of URLs from a finite amount of content, most often a date calendar with next-month links that never end, or a faceted filter where every combination of options has its own address. The crawler keeps finding new URLs, spends its entire budget there, and never reaches the pages that matter.

Does JavaScript navigation break crawling?

It can, and the failure is uneven. Googlebot renders JavaScript and will usually find the links eventually, on a delayed second pass. Most AI crawlers do not render at all, so a menu built from click handlers rather than anchor elements is invisible to them. A real href is the fix and costs nothing.

How do I find dead ends on my own site?

Look for pages with no outbound internal links in the body, which is the structural definition. In practice they cluster: the last page of a paginated series, a detail page whose only links are in the header, a landing page built outside the template, and anything behind a form submission.

The routes a bot takes, and where each one stops

The wall is never at the page you are worried about. It is two hops in front of it, in a navigation component nobody has looked at, and everything behind it is invisible regardless of how good it is.

The short answer

A crawl path is the route a bot takes from your entry point to a page, and it stops at dead ends and traps. Dead ends are pages with no onward links, JavaScript-only navigation and gated sections; traps are infinite calendars and filter combinations that generate URLs forever. One page fetching successfully proves nothing about the route to it.

CriticalAudit check · Crawl paths

Where the walking stops

A crawl is a walk. Something starts at a URL, reads the links, follows them, reads those links, and continues. A path is one of those routes, and the interesting property of a path is not its length but its terminus: the point after which nothing further is discovered.

Two of these three routes stop before they arrive. Neither of the walls is on the pages that are lost, which is what makes this class of problem so hard to diagnose from a report about those pages: they have no findings at all, because nothing ever read them.

This is why the check reports routes rather than pages. A finding that says "412 product pages are missing" sends you to look at product pages, where you will find nothing wrong. A finding that says "the route through the listing stops at page two" sends you to the one component that is actually broken.

Six dead ends and two traps

A route can fail by stopping and by never stopping. Both remove the pages behind it, for different reasons.

The dead ends:

Navigation built in script. A menu, a listing or a next control that exists only after the page runs. There is no href in the served markup, so there is no link to follow. This is the most common cause by a wide margin, and the rendering guide covers what a crawler actually receives.
Infinite scroll with no paged fallback. Content loads as you scroll, which is not a thing a crawler does. Without numbered pages behind the same data, everything past the first screen is unreachable.
Anything behind a form. A search box, a postcode lookup, a filter that submits. A crawler does not fill in forms, so a catalogue reachable only by searching it is not reachable.
A session or auth requirement mid-route. A section that redirects anonymous visitors to a login. The route ends at the login page, which usually returns a 200 and looks like a successful crawl.
A nofollow on the only route. The link exists and declines to pass. Where a section is reachable through exactly one link and that link is marked nofollow, the section has no route.
An error or a block partway. A 500 on a hub page, a 403 from a firewall rule, a rate limit reached during the crawl. The pages behind an intermittent failure are discovered intermittently, which is worse than a clean break because it looks like noise.

The traps are the opposite failure. A route that generates new URLs endlessly consumes all the attention that would have gone to your real pages.

Calendars and date navigation. A next-month link with no end produces an infinite series of empty pages, and crawlers will keep following it for a surprisingly long time.
Faceted navigation with combinable filters. Three filters with ten values each is a thousand URLs, and adding a sort order doubles it. Every one is crawlable if you link to it, and most of them show the same items in a different order, which makes this a duplication problem at the same time.

One fetch is a step, not a route

A path is a sequence: fetch, extract links, follow, repeat. A single-page audit performs the first step of that and stops, which is enough to tell you what a page contains and not whether anything would have arrived at it.

This one needs the full crawl

A path is a sequence of links followed from a starting page, so it only exists once something has done the walking. A single fetch is one step of a route rather than the route.

The instant search on this site audits a single page, so rather than show you a verdict it cannot support, this guide sends you to the place the check actually runs.

Run the full audit, free

The related question of whether a bot is permitted to fetch a page it does reach is a separate check. Permission and topology fail independently, and a site can have either problem without the other.

Does this affect AI search?

More than almost anything else in this pillar, because the agents involved walk far less than a search crawler does. A search engine will eventually find a page through an obscure route. Something fetching two or three pages to answer a question will not, and a route that dead-ends at hop two is a route it never gets past.

The script-rendered navigation problem is the sharpest version. Many agents fetch a URL and read the markup without executing anything, which means a site whose entire navigation is client-side presents as a home page with no links on it. Search crawlers do render, with delay and with limits. The other consumers frequently do not, so this is a case where the gap between the two audiences is the whole finding.

The traps matter here too, in a way that is easy to miss. An agent with a small budget that follows a calendar link has spent that budget on empty pages. Limiting what is linkable is not only about crawl efficiency for search; it decides what a brief visit sees.

Why a blocked route outranks a buried page

Depth is a tax and a dead end is a wall. Critical is for findings where pages are not in the running at all, and this is the purest example: the pages behind a broken route are not slow to be crawled or ranked poorly, they are absent.

The rating is also about how many pages one defect removes. A single component, one listing template, one menu, one filter panel, sits in front of an entire section. There is no other check in this audit where one line of code reliably removes several hundred pages, and none where the removed pages look so healthy when you inspect them individually.

The last reason is diagnostic difficulty. Every other Critical finding shows up as a problem on the page that has it. This one shows up as an absence, and an absence is invisible in a report that lists what was found. Ratings exist partly to direct attention, and this is the finding least likely to attract it on its own.

Walking one route by hand

Pick the section you care most about and follow it the way a crawler would: from the served HTML, with no JavaScript, three hops deep.

walk.sh

HOME=https://example.com

hop() { curl -s "$1" | grep -o 'href="[^"#]*"' | sed 's/href="//;s/"$//' | sort -u; }

echo "--- depth 1, from the home page"
hop "$HOME" | head -40

echo "--- depth 2, from the listing"
hop "$HOME/shop" | head -40

echo "--- is there a link to page 2 at all?"
curl -s "$HOME/shop" | grep -oE 'href="[^"]*(page=|/page/)[^"]*"' | sort -u

The third command is the one that finds the most common dead end on the internet. If it prints nothing and your listing visibly has pagination, that pagination is not in the HTML, and every page of the series after the first has no route.

Then check the traps from the other direction.

Look in your server logs for the most-requested URL patterns from crawler user agents. If the top of that list is filter combinations, sort orders or calendar months, you have found where your crawl budget is going, and the fix is to stop linking to those URLs rather than to add a canonical tag to them.

Linking is the permission that matters. A URL that exists but is not linked from anywhere is not a trap; it is just an address. The trap is created by the link.

Questions this check raises

What is a crawler trap?: A pattern that generates an unbounded number of URLs from a finite amount of content, most often a date calendar with next-month links that never end, or a faceted filter where every combination of options has its own address. The crawler keeps finding new URLs, spends its entire budget there, and never reaches the pages that matter.
Does JavaScript navigation break crawling?: It can, and the failure is uneven. Googlebot renders JavaScript and will usually find the links eventually, on a delayed second pass. Most AI crawlers do not render at all, so a menu built from click handlers rather than anchor elements is invisible to them. A real href is the fix and costs nothing.
How do I find dead ends on my own site?: Look for pages with no outbound internal links in the body, which is the structural definition. In practice they cluster: the last page of a paginated series, a detail page whose only links are in the header, a landing page built outside the template, and anything behind a form submission.