What is the difference between crawling and indexing?

When a product page does not show up on Google, the usual line is "Google can't see it." Behind that line are two different failures, with different fixes: the crawler never gets to request the page, or it requests it and Google decides not to store it in its index. Each phase has its own entry, the crawler's and the one on indexing; what is useful is the border between them: the cases where one happens without the other and how to tell which one you are in. For a store, knowing which side the problem is on decides who fixes it and what gets touched: the server and its access rules in one case, the page and its signals in the other.

How it works

Google describes Search in three phases. In crawling, its crawlers download the text, images and videos of the pages they have found; in indexing, Google analyzes that content and stores the information in its index; in the third, when someone searches, it returns what it considers relevant.⁠^{[1]Google Search Central. In-depth guide to how Google Search works} Between the first two there is a step that is often forgotten: pages that respond with a 200 status go into a rendering queue, where a headless Chromium runs the JavaScript, and what gets indexed is the rendered HTML.⁠^{[2]Google Search Central. Understand JavaScript SEO basics}

Each phase has its own obstacles, and Google lists them separately. Crawling is held back by server problems, network problems and robots.txt rules that do not let Googlebot through. Indexing is held back by low-quality content, robots rules on the page itself that forbid indexing it and a site design that makes it hard.⁠^{[1]Google Search Central. In-depth guide to how Google Search works}

And each phase has its own switch. robots.txt acts before the visit exists: if a URL is blocked, Googlebot does not even make the HTTP request.⁠^{[2]Google Search Central. Understand JavaScript SEO basics} The noindex rule acts afterward, because it travels inside the response: Google finds it when it tries to index the page, and that is why it asks you to remove the robots.txt block when what you want is for a page not to be indexed.⁠^{[3]Search Console Help. Page indexing report}

Search Console's two reports separate the two phases. Crawl Stats, in the property settings, shows Google's crawl history on your site: requests, downloaded size, average response time, host status and breakdowns by response code, file type, crawl purpose and Googlebot type. Google aims it at advanced users: if your site has fewer than a thousand pages, it says you should not need it.⁠^{[4]Search Console Help. Crawl Stats report} The page indexing report says, URL by URL, what is in the index and why the rest is not.⁠^{[3]Search Console Help. Page indexing report}

When one happens without the other

The statuses in the page indexing report, read calmly, are a catalog of those combinations.⁠^{[3]Search Console Help. Page indexing report}

Crawled and not indexed, because Google decides. "Crawled - currently not indexed" means Google has crawled the page but has not indexed it, and that it may index it later.⁠^{[3]Search Console Help. Page indexing report} The crawl worked: what fails is in what the page offers compared with the others, and asking for more visits does not fix it.

Crawled and not indexed, because you ask for it. "URL marked 'noindex'": when trying to index the page, Google found the directive. "Page with redirect": it is a non-canonical page that redirects to another and will not be indexed; the target may or may not be indexed, depending on what Google thinks.⁠^{[3]Search Console Help. Page indexing report} In both cases the visit is indispensable, because without it there is no noindex or 301 redirect to read.

Indexed without having been crawled. "Indexed, though blocked by robots.txt." Google does not request the page, but it can index it with information from another page that links to it, and since it has not read it, the snippets it shows for it are very limited.⁠^{[3]Search Console Help. Page indexing report} The URL is in the index with what others say about it, not with what it says about itself.

Found and not yet visited. "Discovered - currently not indexed" means Google has found the page but has not crawled it, probably because doing so would have overloaded the site and it had to postpone; that is why the report gives no last crawl date.⁠^{[3]Search Console Help. Page indexing report} Here the problem really is one of crawling, and it is where crawl budget comes in.

Indexed, but with an old version. The index stores what Google read on its last crawl. URL Inspection warns of it: what it shows of the indexed version does not correspond to the version published on the web, and the live test is there to see the current one.⁠^{[5]Search Console Help. URL Inspection tool} A price changed this morning is not in the index until Google requests the page again and processes it.

Read with no index behind it. Outside Google, a read may not feed any index. OpenAI explains that ChatGPT-User does not crawl the web automatically and is not used to decide whether content appears in ChatGPT search: these are visits a person initiates, and robots.txt rules may not apply to them.⁠^{[6]OpenAI. Overview of OpenAI Crawlers} A line from it in your logs is a read, not a place in an index.

In a store

A fictitious example: a tea store crosses two sources for five URLs. The first data column comes from the server logs, counting only verified Googlebot requests; the second, from the page indexing report. The status names are the ones in Google's English help.⁠^{[3]Search Console Help. Page indexing report}

Neither source is enough alone. The logs do not know what is in the index, and the indexing report does not show every visit. The third row is the least intuitive: an internal search results page, blocked so it is not crawled and yet in the index because another page links to it.

How to check it

1. Inspect the URL and read crawling and indexing separately. On the indexed version, URL Inspection gives the last time Google crawled the page, the type of crawler, whether your site allowed it to crawl and whether it could fetch it from the server; separately, whether the page refuses indexing.⁠^{[5]Search Console Help. URL Inspection tool} The first speaks to crawling; the last, to indexing.
2. Compare with the live test. "View tested page" shows the rendered screenshot, the HTML, the HTTP headers and the resources loaded. A positive result does not guarantee the page shows up in Search.⁠^{[5]Search Console Help. URL Inspection tool}
3. Sort the reasons in the page indexing report into two piles. Crawling: the robots.txt block, "Server error (5xx)" and "Discovered - currently not indexed". Indexing: "Crawled - currently not indexed", "URL marked 'noindex'", the duplicates and "Page with redirect".⁠^{[3]Search Console Help. Page indexing report}
4. If you are past a thousand pages, open Crawl Stats and look at the breakdowns by response code and the host status.⁠^{[4]Search Console Help. Crawl Stats report} A spike of errors there explains pages that stay in "Discovered".
5. Fix in order: crawling first, then indexing. A noindex behind a robots.txt block is never read.⁠^{[3]Search Console Help. Page indexing report}

Related concept: JavaScript rendering

Rendering is the phase that sits between the two, and the one that confuses most in stores whose themes assemble the page in the browser. Google puts pages that respond with a 200 into the rendering queue and uses the rendered HTML to index; a page with another response code may skip rendering, and a noindex in the original HTML can also cause it to be skipped.⁠^{[2]Google Search Central. Understand JavaScript SEO basics} That is why a page can be well crawled, because the server responded, and still enter the index without the fact that only a script paints if that fact does not reach the rendered HTML. The entry on JavaScript rendering explains how to check what arrives.

To go further, the crawling and indexing reading path orders the rest of the terms, and the guide to what an AI crawler sees in your store takes the same question to the agents that are not Google.