A sitemap is a claim about which pages matter
A sitemap is a shortlist of the pages you want crawled, submitted by you, with nothing else in it. Most sitemaps are an export of everything the CMS can address, which is a different document entirely and a much less useful one.
The short answer
An XML sitemap is a shortlist of the URLs you want indexed, not an inventory of every URL that exists. Four kinds must never appear in it: non-canonical duplicates, noindexed pages, redirecting URLs and error pages. A sitemap contradicting your own directives is worse than no sitemap, because it teaches the crawler to discount it.
A shortlist, not an inventory
The file has two jobs. It helps a crawler discover URLs it might not reach by following links, and it tells you, through your search console, what happened to the URLs you submitted. Both of those only work if the list is a statement of intent.
A sitemap containing every addressable URL, including filters, paginated views, tag archives and redirects, is not making a claim. It is describing the surface area of your application, and the reporting you get back is correspondingly useless: a coverage report where three quarters of the exclusions are pages you never wanted indexed tells you nothing about the quarter that matters.
So the useful discipline is subtractive. The list should be the pages you would be annoyed to find missing from a search result, and nothing else.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/guides/sharpening</loc>
<lastmod>2026-08-14</lastmod>
</url>
<url>
<loc>https://example.com/guides/carbon-steel</loc>
<lastmod>2026-07-02</lastmod>
</url>
</urlset>Two elements, and that is genuinely all you need. changefreq and priority are in the specification and have been publicly described as ignored by Google for years, so emitting them is decoration. lastmod is used, and only if it is honest: a file where every URL updated today is a file whose dates get discounted, so wire it to real content changes or leave it out.
The other limits worth knowing: 50,000 URLs and 50MB uncompressed per file, and a sitemap index when you need more than one. Declare it in robots.txt with a Sitemap: line, absolute URL, and submit it in your search console so the reporting comes back to you.
Four kinds of URL that must never be in it
Each of these is a direct contradiction: the file says "please index this" and something else on the same site says the opposite. Contradictions are the most common sitemap defect and the easiest to find.
- Anything carrying a noindex. You have submitted a page and told the crawler not to index it. Both instructions are yours, one of them is wrong, and the report you get back will be full of exclusions you caused.
- Anything that redirects. A URL in your sitemap should return a 200. Listing the old address after a migration is asking a crawler to keep visiting a hop, and it is the signature of a sitemap generated before a restructure and never regenerated.
- Anything canonicalised elsewhere. If the page names another URL as canonical, submit the canonical. Submitting the variant asks for the version you have already said is not the one you want.
- Anything blocked in robots.txt. The crawler is not permitted to fetch it, so the submission cannot be acted on. This pairing usually means two people configured two files.
A fifth, less absolute: 404s. They arrive by themselves, because a sitemap generated at build time lists pages that get deleted afterwards. A stale file is normal and a file full of dead URLs is a signal that nothing regenerates it, which devalues everything else in it.
Look for a sitemap on this site
This checks whether a sitemap can be found at all: at the conventional locations, and declared in your robots.txt. That is the precondition rather than the check. Whether your important pages are inside the file is the part that needs the file parsed and compared against a crawl.
No signup required. Each free search audits one page, paste any URL to see it in action.
What a sitemap will not do for you
Three beliefs that survive because the file feels like a submission form.
It is not a substitute for a link. This is the big one. A page listed in a sitemap and linked from nowhere is still an orphan: it may get crawled, and it arrives with no context, no anchor text describing it and no share of anything your internal links distribute. Sitemaps aid discovery, they do not confer importance.
It is not a request that gets granted. Submitting a URL does not mean it will be indexed. Thin pages, near-duplicates and pages with no demand are excluded whether or not they are in the file, and the coverage report telling you so is describing a judgement rather than a bug.
It does not fix a crawl that stops. A sitemap can hand a crawler a hundred URLs, and if the route to those pages dead-ends, everything about how those pages relate to each other is still missing. The file is a list, not a structure.
Where a sitemap genuinely earns its keep: sites whose content is not reachable in a few hops.
A large archive, a catalogue with deep pagination, a site where the useful pages are behind a filter interface. There the file is doing real work, because link discovery alone would take a long time to arrive. On a two-hundred-page site with decent navigation, the sitemap is mostly valuable for the reporting it unlocks rather than for the discovery.
The other genuinely useful case is a news or frequently-updated site, where an accurate lastmod is the cheapest way to say what changed since the last visit.
Does this affect AI search?
Less than it affects search, and in a narrower way. Nothing about being in a sitemap makes a page more likely to be retrieved or quoted, and there is no equivalent of a coverage report on the other side.
Where it matters is discovery, which is a real problem for agents that fetch far less comprehensively than a search crawler does. A page nothing links to and no file lists is a page an occasional visitor has no route to at all. The sitemap is one of the few things you control that makes a URL findable without a link.
There is also an adjacent file worth mentioning here rather than confusing with this one. An llms.txt is a curated description of what your site is and where the substance lives, aimed specifically at model consumers. It is a different document with a different audience, and one is not a version of the other.
Why a missing sitemap is Important and not Critical
Because a site with good internal linking is crawled thoroughly without one. Discovery through links is the primary mechanism and it works. Nothing is excluded by the absence of this file, which is what keeps it below the top of the scale.
It rates Important for two reasons that have nothing to do with crawling. It is the cheapest file on this list to produce, usually one line of configuration in any modern framework, so the cost of the finding is close to zero. And submitting it is what turns your search console from a set of aggregate charts into a per-URL report on the pages you nominated, which is the single most useful diagnostic available to a site owner.
The rating rises in practice on large sites. Past a few thousand URLs, discovery genuinely does not reach everything, and past that point the file stops being a convenience. This guide is where that distinction gets made, since the chip cannot vary by site size.
Checking a sitemap against itself
Every defect in the list above is findable in one pass, and the pass is worth automating because a generated file drifts.
curl -s https://example.com/sitemap.xml \
| grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' | sort -u > /tmp/urls.txt
printf 'submitted: %s URLs\n' "$(wc -l < /tmp/urls.txt)"
while read -r url; do
head=$(curl -s -I -o /dev/null -w '%{http_code}' "$url")
body=$(curl -s "$url" | tr '\n' ' ')
case "$head" in
200) ;;
3*) echo "redirects ($head): $url" ; continue ;;
*) echo "not 200 ($head): $url" ; continue ;;
esac
printf '%s' "$body" | grep -qi 'name="robots"[^>]*noindex' \
&& echo "noindex but submitted: $url"
canon=$(printf '%s' "$body" \
| grep -o 'rel="canonical" href="[^"]*"' | sed 's/.*href="//;s/"$//')
[ -n "$canon" ] && [ "$canon" != "$url" ] \
&& echo "canonical points elsewhere: $url -> $canon"
done < /tmp/urls.txtAnything this prints is a contradiction between two files you control, which makes it the least arguable class of finding in an audit. Run it after any migration, because that is when a sitemap and a set of redirects most reliably disagree.
Questions this check raises
- What should not be in an XML sitemap?
- Anything you are not asking to have indexed: URLs that redirect, URLs returning an error, pages carrying a noindex directive, and any URL that canonicalises to a different address. Each of those is a contradiction between two signals from the same site, and the recorded response is that the sitemap becomes less trusted overall.
- Does a sitemap help a page rank?
- No. It helps a page get discovered, which matters on a large site, a new site, or a site with weak internal linking. Once a URL is known and crawled regularly, its presence in a sitemap changes nothing about how it ranks. Discovery and ranking are separate problems.
- How accurate does lastmod need to be?
- Accurate enough to be believed. A lastmod that advances on every deploy tells the crawler that every URL changed at once, which is obviously false, and the field then gets ignored for that site. Once ignored, telling the truth later does not restore it, so the safe options are a real date or no date at all.