The robots directives that are not noindex

Every directive above can be delivered as an X-Robots-Tag response header instead of a meta tag. Same tokens, same meanings, different transport.

# a PDF, which has no head to put a meta tag in
X-Robots-Tag: noindex

# targeting one crawler, and allowing a capped snippet for others
X-Robots-Tag: googlebot: noindex, nofollow
X-Robots-Tag: max-snippet:120

# nginx, applied to a directory of documents
location /reports/ {
  add_header X-Robots-Tag "noindex" always;
}

You need the header form in three situations, and only three. For files that are not HTML, which is the main one: a PDF, a spreadsheet, an image, a plain text file. For directives applied across a path pattern, where setting it at the server is one rule rather than a template change. And for crawler-specific instructions, since the header accepts a user agent prefix that the meta tag cannot express.

A page can carry a meta tag and a header at once, and both can name several tokens. The rule is that the most restrictive instruction wins. A noindex in a header and an index in a meta tag resolves to noindex, and there is no notion of one overriding the other by being later or more specific.

Which produces a practical asymmetry worth internalising: adding a permissive directive can never undo a restrictive one somewhere else. Teams try this. Putting index, follow in a template does nothing to rescue pages that a CDN rule is marking noindex, and it creates the impression that the markup has been checked.

Two more resolution details. A directive naming a specific crawler beats the generic one for that crawler, which is how the header form targets one agent. And a page blocked in robots.txt has its directives never read at all, so a noindex behind a disallow is a message nobody receives, leaving the URL eligible to appear with no description. That combination is covered properly in the indexability guide.

Because the rating has to cover the worst thing the check finds, and the worst thing here removes pages from search entirely. Most of the vocabulary is a preference about a thumbnail or a snippet length. One token, in a header nobody can see, silently deindexes a section, and Critical is the rating for findings that take a page out of the running.

The invisibility is what settles it. Every other Critical finding leaves evidence on the page: a missing title, a broken link, a canonical pointing somewhere odd. A response header leaves nothing in the markup, nothing in your repository and nothing in a browser window. A check that reads them is doing something a person cannot do by looking.

The corollary is that most instances of this finding will be uninteresting, and that is expected. A page declaring index, follow is telling you the default twice. Read the report for the tokens that give something up, and read the header before you conclude the markup is the whole story.

Both sources, one command, because checking only the markup is how this gets missed.

URL=https://example.com/reports/q2-2026

# the header form, which is invisible in a browser
curl -s -I "$URL" | grep -i 'x-robots-tag'

# the meta tag
curl -s "$URL" | grep -io '<meta[^>]*name="robots"[^>]*>'

# non-HTML files can only use the header, so check them directly
curl -s -I https://example.com/files/price-list.pdf \
  | grep -iE '^(HTTP/|content-type|x-robots-tag)'

# and as a specific crawler, since the header can target one
curl -s -I -A 'Googlebot' "$URL" | grep -i 'x-robots-tag'

The last command matters more than it looks. A header applied conditionally by user agent will not appear in an ordinary request, so a check that only runs as your browser can report a clean page that is being told something else.

When a directive turns out to be arriving in a header you did not write, the place to look is your CDN or reverse proxy configuration rather than your application, and the change that introduced it is usually months old.