# Crawling vs Indexing: The Difference, Case by Case

> Crawling and indexing differ: when Google reads a page without storing it, stores it without reading it, and how to tell which side your problem is on.

URL: https://clione.ai/glossary/crawling-vs-indexing/
Lang: en

When a [product page](/glossary/product-page/) does not show up on Google, the usual line is "Google can't see it." Behind that line are two different failures, with different fixes: the [crawler](/glossary/web-crawler/) never gets to request the page, or it requests it and Google decides not to store it in its index. Each phase has its own entry, the crawler's and the one on [indexing](/glossary/indexing/); what is useful is the border between them: the cases where one happens without the other and how to tell which one you are in. For a store, **knowing which side the problem is on decides who fixes it and what gets touched**: the server and its access rules in one case, the page and its signals in the other.

## How it works

Google describes Search in three phases. In crawling, its crawlers download the text, images and videos of the pages they have found; in indexing, Google analyzes that content and stores the information in its index; in the third, when someone searches, it returns what it considers relevant.[^google-how-search] Between the first two there is a step that is often forgotten: pages that respond with a 200 status go into a rendering queue, where a headless Chromium runs the JavaScript, and **what gets indexed is the rendered HTML**.[^google-js-seo]

Each phase has its own obstacles, and Google lists them separately. Crawling is held back by server problems, network problems and [robots.txt](/glossary/robots-txt/) rules that do not let [Googlebot](/glossary/googlebot/) through. Indexing is held back by low-quality content, robots rules on the page itself that forbid indexing it and a site design that makes it hard.[^google-how-search]

And each phase has its own switch. `robots.txt` acts before the visit exists: if a URL is blocked, **Googlebot does not even make the HTTP request**.[^google-js-seo] The [noindex](/glossary/noindex/) rule acts afterward, because it travels inside the response: Google finds it when it tries to index the page, and that is why it asks you to remove the `robots.txt` block when what you want is for a page not to be indexed.[^gsc-page-indexing]

|                                                                        | Crawling                                                         | Indexing                                                                               |
| ---------------------------------------------------------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| What happens                                                           | A program requests the URL and downloads the response            | Google analyzes what was downloaded and decides whether to store it in the index       |
| What stops it                                                          | Server or network errors, `robots.txt` rules[^google-how-search] | Low quality, robots rules on the page, a design that makes it hard[^google-how-search] |
| What controls it                                                       | `robots.txt`                                                     | `noindex` on the page or in its headers                                                |
| Where you see it in [Search Console](/glossary/google-search-console/) | The Crawl Stats report[^gsc-crawl-stats]                         | The page indexing report[^gsc-page-indexing]                                           |
| Where the fix usually is                                               | In the server, the CDN and `robots.txt`                          | In the page and the signals it declares                                                |

Search Console's two reports separate the two phases. Crawl Stats, in the property settings, shows Google's crawl history on your site: requests, downloaded size, average response time, host status and breakdowns by response code, file type, crawl purpose and Googlebot type. Google aims it at advanced users: **if your site has fewer than a thousand pages, it says you should not need it**.[^gsc-crawl-stats] The page indexing report says, URL by URL, what is in the index and why the rest is not.[^gsc-page-indexing]

## When one happens without the other

The statuses in the page indexing report, read calmly, are a catalog of those combinations.[^gsc-page-indexing]

**Crawled and not indexed, because Google decides.** "Crawled - currently not indexed" means Google has crawled the page but has not indexed it, and that it may index it later.[^gsc-page-indexing] The crawl worked: what fails is in what the page offers compared with the others, and asking for more visits does not fix it.

**Crawled and not indexed, because you ask for it.** "URL marked 'noindex'": when trying to index the page, Google found the directive. "Page with redirect": it is a non-canonical page that redirects to another and will not be indexed; the target may or may not be indexed, depending on what Google thinks.[^gsc-page-indexing] In both cases the visit is indispensable, because without it there is no `noindex` or [301 redirect](/glossary/301-redirect/) to read.

**Indexed without having been crawled.** "Indexed, though blocked by robots.txt." Google does not request the page, but it can index it with information from another page that links to it, and since it has not read it, **the snippets it shows for it are very limited**.[^gsc-page-indexing] The URL is in the index with what others say about it, not with what it says about itself.

**Found and not yet visited.** "Discovered - currently not indexed" means Google has found the page but has not crawled it, probably because doing so would have overloaded the site and it had to postpone; that is why the report gives no last crawl date.[^gsc-page-indexing] Here the problem really is one of crawling, and it is where [crawl budget](/glossary/crawl-budget/) comes in.

**Indexed, but with an old version.** The index stores what Google read on its last crawl. URL Inspection warns of it: what it shows of the indexed version does not correspond to the version published on the web, and the live test is there to see the current one.[^gsc-url-inspection] A price changed this morning is not in the index until Google requests the page again and processes it.

**Read with no index behind it.** Outside Google, a read may not feed any index. OpenAI explains that [ChatGPT-User](/glossary/chatgpt-user/) does not crawl the web automatically and is not used to decide whether content appears in [ChatGPT search](/glossary/chatgpt-search/): these are visits a person initiates, and `robots.txt` rules may not apply to them.[^openai-bots] A line from it in your logs is a read, not a place in an index.

## In a store

A **fictitious example**: a tea store crosses two sources for five URLs. The first data column comes from the server logs, counting only verified Googlebot requests; the second, from the page indexing report. The status names are the ones in Google's English help.[^gsc-page-indexing]

| URL (fictitious example)         | Does Googlebot request it? | Status in Search Console              | Side of the problem     | What to do                                                                         |
| -------------------------------- | -------------------------- | ------------------------------------- | ----------------------- | ---------------------------------------------------------------------------------- |
| `/products/cast-iron-teapot-06l` | Yes, every week            | Crawled - currently not indexed       | Indexing                | Improve the page against the similar ones; more crawling changes nothing           |
| `/products/sencha-100g`          | No                         | Discovered - currently not indexed    | Crawling                | Check server errors and response time; link it from pages that are already crawled |
| `/search?q=matcha`               | No                         | Indexed, though blocked by robots.txt | Both                    | Remove the block and use `noindex`, as Google asks[^gsc-page-indexing]             |
| `/products/gift-set-2025`        | Yes                        | Page with redirect                    | Neither, it is expected | Check that the destination is equivalent and indexed                               |
| `/pages/draft-sale`              | Yes                        | URL marked 'noindex'                  | Neither, if intentional | Remove the `noindex` the day the page has to show up on Google                     |

**Neither source is enough alone.** The logs do not know what is in the index, and the indexing report does not show every visit. The third row is the least intuitive: an internal search results page, blocked so it is not crawled and yet in the index because another page links to it.


**Search Console's removals tool does not take a URL out of the index: it hides it.** A temporary removal request keeps the URL from showing in results for about six months, but **it does not stop Google from continuing to crawl it**. To remove it permanently you have to delete the content and return a 404 or a 410, protect it with a password or use `noindex`. If you do not, the page can be crawled again and reappear when the block ends.[^gsc-removals]


## How to check it

1. **Inspect the URL and read crawling and indexing separately.** On the indexed version, URL Inspection gives the last time Google crawled the page, the type of crawler, whether your site allowed it to crawl and whether it could fetch it from the server; separately, whether the page refuses indexing.[^gsc-url-inspection] The first speaks to crawling; the last, to indexing.
2. **Compare with the live test.** "View tested page" shows the rendered screenshot, the HTML, the HTTP headers and the resources loaded. A positive result does not guarantee the page shows up in Search.[^gsc-url-inspection]
3. **Sort the reasons in the page indexing report into two piles.** Crawling: the `robots.txt` block, "Server error (5xx)" and "Discovered - currently not indexed". Indexing: "Crawled - currently not indexed", "URL marked 'noindex'", the duplicates and "Page with redirect".[^gsc-page-indexing]
4. **If you are past a thousand pages, open Crawl Stats** and look at the breakdowns by response code and the host status.[^gsc-crawl-stats] A spike of errors there explains pages that stay in "Discovered".
5. **Fix in order: crawling first, then indexing.** A `noindex` behind a `robots.txt` block is never read.[^gsc-page-indexing]

## Related concept: JavaScript rendering

Rendering is the phase that sits between the two, and the one that confuses most in stores whose themes assemble the page in the browser. Google puts pages that respond with a 200 into the rendering queue and uses the rendered HTML to index; a page with another response code may skip rendering, and a `noindex` in the original HTML can also cause it to be skipped.[^google-js-seo] That is why a page can be well crawled, because the server responded, and still enter the index without the fact that only a script paints if that fact does not reach the rendered HTML. The entry on [JavaScript rendering](/glossary/javascript-rendering/) explains how to check what arrives.

To go further, the [crawling and indexing reading path](/learn/crawling-and-indexing/) orders the rest of the terms, and the guide to [what an AI crawler sees in your store](/guides/what-an-ai-crawler-sees/) takes the same question to the agents that are not Google.