Reading a depth distribution instead of a depth number

The expectation has to scale, and this is where generic advice does the most damage. A site cannot hold a hundred thousand pages within three clicks of its home page unless every page links to thousands of others, which no usable interface does.

The rough arithmetic is worth doing once. If each page links to around 100 others, depth one reaches 100, depth two reaches 10,000 and depth three reaches a million. That is the theoretical ceiling and reality is far below it, because links overlap heavily and most navigation links to the same places. A practical figure is that each level multiplies reach by somewhere between five and thirty.

Which gives you a way to judge your own chart rather than a rule to obey:

• A few hundred pages should be mostly within three hops. If a two-hundred-page site has a tail at depth seven, that is not scale, it is a structural accident.
• A few thousand should peak at three or four. A long, thin tail here usually means the only route into older content is chronological.
• Past a hundred thousand, depths of six and seven are normal. The question stops being whether pages are deep and becomes whether the pages that earn money are.

Then compare the shape against your own priorities rather than against a benchmark. Overlay which depths hold your revenue pages. A distribution that looks poor but keeps every commercially important URL within two hops is a better site than a tidy chart where the products sit at five.

Indirectly, and the honest framing is about which parts of your site are effectively public. Agents that fetch a small number of pages see the shallow end of your distribution and nothing else. So the shape of the chart is, roughly, a map of what is available to be quoted.

The tail is where this matters most, and it is often where the useful material is. Older documentation, archived reference pages, the detailed answers that accumulated over years: these are exactly the pages that end up deep, because the routes to them are chronological or paginated. A site can have the best answer to a question sitting at depth eight, unreachable by anything that only follows a couple of links.

Which suggests a specific use for this chart that has nothing to do with crawl budget. Find the depth of your ten best answers. If any of them are past four, the fix is a link from somewhere prominent, and it is worth more than anything you could do to the page itself.

A real distribution needs a crawl. A decent approximation needs your sitemap and the observation that path segments usually track depth on a well-structured site.

curl -s https://example.com/sitemap.xml \
  | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' \
  | sed 's|https\?://[^/]*||' \
  | awk -F/ '{ print NF - 1 }' \
  | sort -n | uniq -c \
  | awk '{ printf "%d segments: %5d  ", $2, $1
           for (i = 0; i < $1 / 20; i++) printf "#"
           print "" }'

# and the deep end, listed, because the pattern is the finding
curl -s https://example.com/sitemap.xml \
  | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' \
  | awk -F/ 'NF - 3 >= 5' | head -20

Path depth is not click depth, and the difference is the point: a flat URL can be deep in the graph and a nested one can be one click away. Treat the output as a way of finding candidates, then check the ones that look wrong with the per-page walk.

The second command is the more useful of the two. Read the twenty deepest paths and see whether they have something in common. They almost always do, and that commonality is the template to fix.