prebaked.
SEO reference

robots.txt

Last reviewed

robots.txt is a plain text file at the root of a host that tells crawlers which URL paths they may fetch. It controls crawling, not indexing, and it is a request to well-behaved crawlers, not access control.

Syntax

The file lives at /robots.txt (all lowercase) in the top-level path. It is made of groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules.

# Block one folder for all crawlers
User-agent: *
Disallow: /admin/
Allow: /admin/help/

# Block everything for one crawler
User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml
Field Value Meaning
User-agent Product token, or * for all Which crawler the group applies to. Matching is case-insensitive.
Disallow Path starting with / Do not fetch URLs that start with this path. An empty value disallows nothing.
Allow Path starting with / Fetching is allowed. Used to open a subpath inside a disallowed path.
Sitemap Absolute URL Location of a sitemap. Not tied to any group. Google supports it; RFC 9309 allows such extra records.
* in a path Wildcard Matches zero or more characters.
$ in a path End anchor Marks the end of the URL.

Google supports user-agent, allow, disallow and sitemap. Other fields such as crawl-delay are not supported by Google. Anthropic says it supports Crawl-delay for its bots.

How rules are matched

  • One group per crawler. A crawler uses the group with its own token. If several groups name the same token, the rules are combined into one group. If none matches, the * group applies.
  • Longest match wins. When several rules match a URL, the rule with the longest path applies.
  • Ties go to Allow. If an Allow and a Disallow rule are equivalent, the Allow rule is used.
  • Paths are case-sensitive. Disallow: /fish does not block /Fish.asp.
User-agent: *
# Block every URL with a query string
Disallow: /*?
# Block all PDF files
Disallow: /*.pdf$

Disallow examples

Goal Rule
Block the whole site Disallow: /
Allow the whole site Disallow: (empty) or no rule
Block one folder Disallow: /private/
Block one file Disallow: /private/report.html
Block a file type Disallow: /*.pdf$
Block URLs with parameters Disallow: /*?

Scope, status codes and caching

Situation What happens Source
File location Applies only to the protocol, host and port where it is served. Each subdomain needs its own file. Files in subdirectories are ignored. Google
4xx response (except 429) Treated as if no robots.txt exists. Crawling is not restricted. Google, RFC 9309
5xx or network error Google stops crawling for 12 hours while retrying, then uses the last cached copy for up to 30 days. RFC 9309 says to assume complete disallow. Google, RFC 9309
Caching Google caches the file for up to 24 hours, longer if refreshing fails. Google, RFC 9309
Size Google ignores content after 500 KiB. Google, RFC 9309
Redirects Crawlers should follow at least five redirects. RFC 9309

AI crawler user agent tokens

Token Operator Purpose Follows robots.txt
GPTBot OpenAI Content that may be used to train foundation models Yes
OAI-SearchBot OpenAI ChatGPT search results Yes. Opted-out sites are not shown in ChatGPT search answers.
ChatGPT-User OpenAI Fetches made for a user's request in ChatGPT May not apply, because the fetch is user-initiated
ClaudeBot Anthropic Content that may be used for training Yes
Claude-SearchBot Anthropic Search result quality Yes
Claude-User Anthropic Fetches made for a Claude user's question Yes, per Anthropic's general statement
PerplexityBot Perplexity Perplexity search results. Not used for training. Yes
Perplexity-User Perplexity Fetches made for a user's request Generally ignores robots.txt
Google-Extended Google Use of content for Gemini training and grounding. Not a separate crawler. Yes. Does not affect Google Search inclusion or ranking. 
# Keep content out of AI training, stay visible in AI search
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Blocking a search token (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes your pages from that assistant's search results. Blocking a training token does not.

Common mistakes

  • Using Disallow to remove a page from Google. A disallowed URL can still be indexed without a snippet if other pages link to it. Use noindex and let the page be crawled.
  • Blocking a page that carries noindex. If the crawler cannot fetch the page, it never sees the noindex.
  • Blocking JavaScript and CSS files. Google needs them to render the page.
  • Using robots.txt for canonicalization. Google says not to. Use a canonical tag.
  • Treating it as security. The file is public and lists the paths you want hidden. Use authentication.
  • One file for all subdomains. blog.example.com needs its own /robots.txt.
  • Wrong case in paths. /Admin/ and /admin/ are different paths.

How to check your robots.txt

  1. Open https://www.<your-domain>/robots.txt and confirm it returns 200 and plain text.
  2. Check the status code from the command line:
curl -sI https://www.<your-domain>/robots.txt | head -1
  1. In Google Search Console, the robots.txt report shows which files Google found, when it last fetched them and any parse errors.
  2. Test a single URL against your rules in the URL Inspection tool.

On JavaScript sites

robots.txt is fetched as a static file, so JavaScript plays no part in it. Two things still matter:

  • Do not block your bundles. If Disallow: /static/ or Disallow: /*.js$ blocks the files that build your page, Google cannot render it.
  • Allowing a crawler is not enough. A JavaScript site can allow GPTBot, ClaudeBot and PerplexityBot and still show them an empty page, because these crawlers do not run JavaScript. They only read the HTML your server sends. A prerender layer gives them the finished page.

See what ChatGPT sees on your site

FAQ

Where does robots.txt go?

At the root of each host, for example https://www.example.com/robots.txt. It applies only to that protocol, host and port.

Does Disallow stop a page from being indexed?

No. It stops crawling. Google can still index the URL without its content if other pages link to it. Use noindex to keep a page out of results.

Does Google support crawl-delay?

No. Google ignores crawl-delay. Anthropic says its bots support it.

How do I block ChatGPT in robots.txt?

Add a group for GPTBot to opt out of training and OAI-SearchBot to opt out of ChatGPT search, each with Disallow: /. ChatGPT-User acts on user requests, and OpenAI says robots.txt rules may not apply to it.

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.

What happens if robots.txt returns 404?

Crawlers treat it as if there were no robots.txt and may crawl everything.