Accidental noindex: finding pages telling Google to drop them
Of everything an audit can find, this is the one that has ended careers. A single directive, left behind after a staging launch or inherited from a template, removes pages from search entirely. There is no error, no warning, and no drop in anything you monitor until the traffic goes.
The short answer
A noindex directive removes a page from search results entirely, and it is most often left behind by accident: a staging flag shipped to production, a CMS setting on a template, or a plugin default. It can arrive from a meta tag or an X-Robots-Tag header, and blocking the page in robots.txt afterwards makes it worse, not better.
The directive, and why it is so easy to leave behind
noindex tells a search engine to drop the page from its index. Not to rank it lower, not to crawl it less: to remove it. It is honoured quickly and completely, which is exactly what you want during development and exactly what makes it dangerous afterwards.
The reason it survives launches is that it is invisible in every place people look. The page renders normally. It returns 200. It appears in your CMS, your analytics and your own site search. Nothing in a browser shows it, because it is instruction rather than content, and nothing in your deploy pipeline is checking for it unless someone deliberately added that check.
Where the directive comes from
Two delivery mechanisms, and the second one is the one that hides. Most people know to look in the HTML. Far fewer think to look at the response headers, where the same instruction can be set at the CDN or the server without appearing in your source at all.
<!-- 1. In the markup, where people look -->
<meta name="robots" content="noindex, nofollow">
<!-- Also this, which many people do not recognise
as the same instruction: -->
<meta name="robots" content="none">
# 2. In the response headers, where they do not.
# Set at the CDN, load balancer or framework, and
# completely invisible in view-source.
X-Robots-Tag: noindex
# Per-crawler variants are legal here too, which is
# how a page ends up indexed by one engine only:
X-Robots-Tag: googlebot: noindexnone is worth calling out because it reads like an absence of directives and means the opposite: it is shorthand for noindex, nofollow. A reviewer skimming a head for the word "noindex" will scroll straight past it.
The framework and platform sources are predictable. WordPress has a "Discourage search engines" checkbox in reading settings that emits this sitewide; a staging clone with it enabled and then promoted to production carries it along. Shopify applies it to some system-generated pages. In Next.js it is one metadata field, which means it can be set in a layout and inherited by every route beneath it.
Which sources are resolved together
The robots directive that applies to each crawled page, from both the meta element and the X-Robots-Tag header, resolved after render. Any value containing noindex or none is reported as a failure on a page that is otherwise indexable, since the combination of a live, linked, 200-returning page and an instruction to drop it is almost always a mistake.
nofollow without noindex is reported separately and less severely: the page can rank, but it passes nothing to the pages it links to, which matters when the page is a hub.
Check one page for a noindex directive
Paste a URL. This reports the robots directive that applies to the page, or confirms there is none, which is the normal state for a page you want found.
No signup required. Each free search audits one page, paste any URL to see it in action.
Why robots.txt makes a noindex worse, not better
This is the most counterintuitive thing on this page and the mistake made most often by people trying hardest to fix the problem. Blocking a page in robots.txt does not help remove it, and if the page is already indexed, blocking it prevents removal.
The mechanism is simple once stated: noindex is an instruction on the page, so a crawler has to fetch the page to see it. Disallow the URL in robots.txt and the crawler never fetches it, never sees the directive, and the URL can persist in the index indefinitely, sometimes shown with no snippet because nothing may be read from it.
Pick one. Never both on the same URL.
To remove a page from search: allow it in robots.txt and serve noindex, then leave both in place until it is gone. To save crawl budget on a page that is not in the index and should not be: disallow it in robots.txt and do not bother with a meta directive it will never read.
The same reasoning explains the canonical conflict. A canonical says "consolidate signals onto this URL" while a noindex says "discard this URL", and which one an engine acts on is not yours to control. If a page should not be indexed, it should not be a canonical target either.
Telling a deliberate noindex from a disaster
Plenty of pages should carry this directive, and the check firing on them is expected. The test is whether the page is somewhere a searcher should be able to arrive.
- Correct: thank-you and confirmation pages. A page that only makes sense after an action is nonsense as a search result, and indexing it also pollutes your conversion tracking.
- Correct: internal search results and filtered views. Effectively infinite, thin, and duplicative of the pages they list.
- Correct: staging and preview hosts. Provided it is applied to the host rather than to the content, which is the distinction that stops it travelling to production.
- Almost always a disaster: templates and layouts. A directive set in a shared layout applies to every route under it. If the count of noindexed pages matches a whole section of your site, look at the template before looking at the pages.
- Always a disaster: the homepage. No legitimate configuration noindexes a live homepage. Treat it as an incident.
The signal to watch is the ratio. A handful of noindexed utility pages is a healthy site. A hundred and forty of them appearing in one crawl is a deploy.
Why deletion beats degradation on this scale
Because the failure is not a degradation, it is a deletion, and it is the fastest-acting defect in the catalogue. Most problems cost you position gradually. This one removes pages within days of being crawled, and it does so without a single error appearing anywhere you would look.
It is also the defect with the worst ratio of cause to effect: one line, applied by inheritance, capable of taking a whole site out of search. That combination of total, silent and sitewide is the definition Critical exists for.
Does this affect AI search?
Less than people assume, and the gap between the two is itself worth understanding, because it is the source of a common and expensive mistake.
noindex governs search indexes. An assistant fetching your page in real time because a user asked about your product is not consulting a search index for that request, so the directive does not stop it reading or quoting the page. Where it does bite is indirect: many retrieval systems draw their candidate set from search-like indexes, and a page absent from those is a page never considered.
The mistake to avoid runs the other way. People reach for noindexto keep AI systems out, which mostly does not work, and reach for robots.txt to manage search, which as described above actively prevents removal. Both tools are being used for the other one's job. If your goal is controlling AI access, that is a robots.txt decision about specific agents, and it has nothing to do with this directive.
Confirming the page is back
Check both sources, not one. View-source for the meta element, and curl -sI https://example.com/page for the header, because a page can be clean in the markup and blocked at the CDN. Then use a fetch-and-render tool that reports what a crawler received rather than what your browser did.
Removal from the index is quick; return is not. Once the directive is gone the page has to be recrawled before it can be reconsidered, and there is no way to make that instant. Request indexing for the important URLs and expect days rather than hours.
Then add the check to your deploy process, because this is the one defect on this list worth automating. A crawl that fails a build when the count of noindexed pages jumps is cheap to set up and pays for itself the first time someone promotes a staging config.
Questions this check raises
- Why is my page not indexed even though robots.txt allows it?
- Check for a noindex directive, in both the meta robots tag and the X-Robots-Tag response header. The header is the one that gets missed, because it is invisible in view-source and is often applied at the CDN or framework level rather than in the template anyone is looking at.
- Should I block a noindexed page in robots.txt as well?
- No, and doing so is actively harmful. Blocking the URL stops crawlers fetching it, which means they never see the noindex directive, which means a page already in the index stays there indefinitely. Let the crawler reach the page, read the directive and drop it, then block it later if you want to save crawl budget.
- How do I tell a deliberate noindex from a mistake?
- By what kind of page it is. Thank-you pages, internal search results, faceted duplicates and staging environments are supposed to carry it. A noindex on a template that renders hundreds of URLs, or on anything with inbound links or traffic history, is almost always an accident that shipped.