Robots meta tag and X-Robots-Tag

The robots meta tag is a <meta name="robots"> element that tells search engines whether to index a page and follow its links. The X-Robots-Tag is the same set of rules sent as an HTTP response header, which also works for PDFs, images and other non-HTML files.

Syntax

<head>
  <!-- All crawlers -->
  <meta name="robots" content="noindex, nofollow">
  <!-- Google only -->
  <meta name="googlebot" content="nosnippet">
</head>

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex
X-Robots-Tag: googlebot: nofollow

In nginx:

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex" always;
}

Combine rules in one comma-separated list or in separate tags. In the header, rules without a user agent apply to all crawlers. Besides robots for all crawlers, Google supports two crawler-specific names: googlebot for text results and googlebot-news for news results. Google ignores other crawler names.

Values

Google no longer uses noarchive, nocache or nositelinkssearchbox.

How rules combine

• Several crawlers. Google uses the sum of the negative rules. nofollow for robots plus noindex for googlebot means Googlebot applies noindex, nofollow.
• Conflicts. The more restrictive rule applies. nosnippet wins over max-snippet:50.

Page-level nofollow vs rel="nofollow"

Google says links with these rel values will generally not be followed, but the target can still be found through sitemaps or links from other sites.

Common mistakes

• noindex on a page blocked in robots.txt. If Google cannot crawl the page, it never sees the rule. The page can still appear in results. Remove the robots.txt block so Google can see the noindex.
• noindex left on after launch. Staging settings copied to production remove the whole site from results.
• Using nofollow to keep a page out of the index. nofollow controls links. Use noindex.
• noindex together with a canonical to another URL. Mixed signals. Pick one: canonical for duplicates, noindex for pages that should not appear at all.
• Meta tag on a PDF. Non-HTML files need the X-Robots-Tag header.
• Changing noindex with JavaScript. See below.

How to check your page

1. Meta tag in the raw HTML:

curl -s https://www.<your-domain>/page/ | grep -io '<meta[^>]*name="\(robots\|googlebot\)"[^>]*>'

2. X-Robots-Tag header:

curl -sI https://www.<your-domain>/page/ | grep -i '^x-robots-tag'

3. In Google Search Console, the URL Inspection tool shows the HTML Googlebot received, and the Page Indexing report lists pages where Google found a noindex rule.

On JavaScript sites

Google says that when it finds noindex, it may skip rendering and JavaScript execution. So a noindex in the original HTML that JavaScript later removes may never be removed from Google's point of view. Google's advice: if you want the page indexed, do not put noindex in the original page code.

The reverse also fails for some crawlers. A single page app that adds noindex with JavaScript for a 404 route works for Google only after rendering. Crawlers that do not run JavaScript, including the major AI crawlers such as GPTBot and ClaudeBot, never see it, because they only read the HTML the server sends.

Put robots rules in the HTML or the HTTP header that every crawler receives. For JavaScript apps, that means server-side rendering, static generation or prerendering, or an X-Robots-Tag set by the server.

See what ChatGPT sees on your site

FAQ

Related

• robots.txt
• Canonical tag
• Open Graph tags
• Heading tags
• What is prerendering
• How AI crawlers see your site