How to get Google to index your site
Last reviewed
To get Google to index your site, make sure Google can find each URL (links and a sitemap), fetch it (no robots.txt block, no server errors), and read real content in it (no noindex, no empty page). Then ask for a recrawl in Search Console. Google says it "doesn't guarantee that it will crawl, index, or serve your page", so the goal is to remove every technical reason not to.
What the Search Console statuses mean
The Page indexing report in Search Console tells you where a URL stopped. The two most common statuses mean different things.
| Status | What Google says | What it means for you |
|---|---|---|
| Discovered - currently not indexed | "The page was found by Google, but not crawled yet." "Typically, Google wanted to crawl the URL but this was expected to overload the site" | Google knows the URL but has not fetched it. The report shows no last crawl date. Look at server speed, errors and how many low-value URLs you expose. |
| Crawled - currently not indexed | "The page was crawled by Google but not indexed." There is "no need to resubmit this URL for crawling." | Google fetched the page and decided not to index it, for now. Look at what the page actually contains, including what is left after rendering. |
| URL marked 'noindex' | "When Google tried to index the page it encountered a 'noindex' directive and therefore did not index it." | Remove the directive if you want the page indexed. |
| URL blocked by robots.txt | "This page was blocked by your site's robots.txt file." | Google cannot crawl it. Note that robots.txt "does not guarantee that the page won't be indexed through some other means." |
| Soft 404 | The page "returns a user-friendly "not found" message but not a 404 HTTP response code" | Return a real 404 for missing pages, or fix the content if the page should exist. |
| Duplicate without user-selected canonical | "This page is a duplicate of another page, although it doesn't indicate a preferred canonical page." | Google picked another URL as canonical. Only a problem if it picked the wrong one. |
| Page with redirect | "This is a non-canonical URL that redirects to another page." | Expected for redirecting URLs. Check the target instead. |
Checklist: why Google is not indexing your site
Work from top to bottom. Each step depends on the one before.
- Can Google find the URL? Google discovers pages when it "extracts a link from a known page to a new page" or when you submit a list of pages. Link to the page from a page that is already indexed and list it in your sitemap.
- Can Google fetch the URL? Google lists "Problems with the server handling the site", "Network issues" and "robots.txt rules preventing Googlebot's access to the page" as common reasons crawling fails.
5xxand429responses "prompt Google's crawlers to temporarily slow down with crawling." - Does the page return 200? All
4xxerrors except429tell Google the content does not exist. A redirect sends Google to another URL. - Is indexing allowed? No
noindexin the robots meta tag or theX-Robots-Tagheader. In the URL Inspection tool, "Indexing allowed?" shows the reason if it is not. - Is the canonical right? URL Inspection shows the user-declared canonical and the Google-selected canonical. If they differ, Google indexes the other URL.
- Is there real content in the page Google renders? This is where JavaScript sites fail. See the next section.
- Is the page worth indexing? Google says "Indexing isn't guaranteed; not every page that Google processes will be indexed." Thin, duplicate and near-empty pages are the usual candidates.
# Status code and robots header for one URL
curl -s -o /dev/null -D - https://www.<your-domain>/pricing | grep -iE "^HTTP|x-robots-tag"
# robots meta tag in the HTML the server sends
curl -s https://www.<your-domain>/pricing | grep -io '<meta[^>]*name="robots"[^>]*>'
The JavaScript reason: an empty page
If your site builds its pages with JavaScript, the HTML your server sends can be an empty shell with one root element and some scripts. Google fetches that HTML first and renders the JavaScript later: "Googlebot queues all pages with a 200 HTTP status code for rendering", and "the page may stay on this queue for a few seconds, but it can take longer than that."
Things that go wrong on JavaScript pages:
- Rendering fails or misses content. If an API call fails or a script is blocked, the rendered page is as empty as the shell. Google can report it as a soft 404 or leave it crawled but not indexed.
noindexin the shell. "When Google encounters the noindex tag, it may skip rendering and JavaScript execution", so removingnoindexwith JavaScript "may not work as expected."- Blocked scripts. "Google Search won't render JavaScript from blocked files or on blocked pages." Check that robots.txt does not disallow your script and API paths.
- Every missing route returns 200. A single page app that returns
index.htmlwith a 200 status for every path creates soft 404s.
How to see what Google rendered: open URL Inspection, click Test live URL and View tested page. It shows the rendered HTML, the screenshot, loaded resources and JavaScript console messages. Full steps are in how to debug rendering problems.
How to fix it for good: send finished HTML from the server. You can use server-side rendering, static generation or prerendering. The differences are in static vs dynamic pages and JavaScript SEO.
See what ChatGPT sees on your site
How to request indexing
- A few URLs: URL Inspection. Inspect the URL and click Request indexing. The page must pass a quick check for immediate errors first. "There is a daily limit to how many index requests you can submit."
- Many URLs: a sitemap. Submit a sitemap in Search Console and keep
<lastmod>accurate. - Wait. Crawling "can take anywhere from a few days to a few weeks", and "requesting a recrawl multiple times for the same URL won't get it crawled any faster."
Google recommends asking for a recrawl only after an important change. For "Crawled - currently not indexed" pages, resubmitting is not the fix. Changing the page is.
When Prebaked is the right choice
Prebaked is for JavaScript sites where crawlers get an empty shell. It is not a fix for thin content, wrong canonicals or robots.txt blocks.
- Add one snippet in front of your site: Cloudflare Worker, Express, nginx, Netlify or Vercel.
- Search engines and AI crawlers get the page rendered in Chromium, with the same content your visitors see. More on AI crawlers.
- Status codes are kept: a page that renders as a 404 returns 404 to crawlers, and a page can set its status with
<meta name="prerender-status-code" content="404">. - If Prebaked is slow or down, crawlers get your original page.
Prebaked does not guarantee indexing or rankings. It makes sure Google receives a page with content in it.
FAQ
What is the difference between crawled and discovered currently not indexed?
Discovered means Google knows the URL but has not fetched it yet, often because crawling was expected to overload the site. Crawled means Google fetched the page and chose not to index it for now. The first points to crawling, the second to the page itself.
Should I request indexing for crawled currently not indexed pages?
Google says there is no need to resubmit these URLs. Change what the page contains, or how it renders, and then request indexing once.
How long does it take Google to index a new page?
Google says crawling can take from a few days to a few weeks. Requesting a recrawl several times for the same URL does not make it faster.
Why is Google not indexing my JavaScript site?
Often because the HTML Google gets has no content until JavaScript runs, and rendering fails, is blocked or returns an error page with a 200 status. Check the rendered HTML in URL Inspection.
Does a sitemap guarantee indexing?
No. A sitemap helps Google discover URLs. Google does not guarantee that it will crawl or index any page.