# What Is Indexing in Google and How to Check It

> What indexing is, how it differs from crawling, why Google does not store all your product pages and how to read its status in Search Console.

URL: https://clione.ai/glossary/indexing/
Lang: en

Here indexing is not the kind that ties a lease to inflation, or a database index: it is what a search engine does with your page after visiting it. For a store, **a [product page](/glossary/product-page/) that is not in the index does not exist for Google**: it does not show up in results, it cannot bring traffic and it cannot serve as a source in its AI answers either. And it is not unusual for that to happen. Google says plainly that it does not guarantee crawling, indexing or serving any page, even if it meets its essential requirements.[^google-how-search-works]

## How it works

Google describes Search in three stages, and **not every page makes it through all three**: crawling, in which a [crawler](/glossary/web-crawler/) downloads the text, images and videos of the pages it finds; indexing, in which it analyzes that content and stores it in the index, a large database; and serving results, when someone searches and Google returns what it considers relevant.[^google-how-search-works]

What happens inside indexing is more involved than the word "store" suggests. Google processes the text and key content tags and attributes, such as the `<title>` element or `alt` attributes, as well as images and videos. It groups the pages it considers duplicates of each other and picks the most representative one in the group as the canonical; from that page it collects signals it later uses when serving results.[^google-how-search-works] That is why **in a store one URL per group gets indexed, not one per variant or per filter**, and Google says so in its report: your goal should be to get the canonical version of every important page indexed, and duplicate or alternate pages should not be indexed.[^gsc-page-indexing]

The URL that ends up in the index is not entirely your call. A redirect and the [canonical](/glossary/canonical-url/) tag are strong signals, and appearing in a sitemap is a weak one, but all three are hints, not orders: Google may pick a different canonical if it thinks it is better.[^google-canonical] The same goes for the sitemap as a discovery route: it helps Google find your URLs, but it does not guarantee that all of them are crawled and indexed.[^google-sitemaps]

JavaScript adds an intermediate stage. Google renders the page with a recent version of Chrome and runs the JavaScript it finds, as your browser would.[^google-how-search-works] Pages wait in a render queue, and **what gets indexed is the rendered HTML**.[^google-js-seo] Two nuances matter in a store. If the page responds with a status code other than 200, for example a 404, rendering may be skipped. And if Google finds a `noindex` in the original HTML, it may skip rendering and running the JavaScript, so removing it later with a script may do nothing.[^google-js-seo] Google still recommends [server-side rendering](/glossary/server-side-rendering/) or prerendering, because it makes the site faster for users and crawlers.[^google-js-seo]

Two instructions decide what gets in and what does not, and they get mixed up often. The [`robots.txt`](/glossary/robots-txt/) file tells crawlers which URLs they can access; it is mostly for avoiding overloading your server and **it is not a mechanism for keeping a page out of Google**: a blocked URL can still be indexed if other sites link to it, and then it shows up without a description.[^google-robots-intro] To keep a page out of the index there is the [noindex](/glossary/noindex/) rule, as a `<meta name="robots">` tag or as an `X-Robots-Tag` HTTP header. For it to work, the page cannot be blocked in `robots.txt`: the crawler has to be able to read it to see the instruction.[^google-noindex]

Indexing is also the gate to Google's AI answers. To appear as a supporting link in [AI Overviews](/glossary/ai-overviews/) or in [Google AI Mode](/glossary/google-ai-mode/), a page has to be indexed and eligible to be shown in Search with a snippet, and Google states there are no additional requirements.[^google-ai-features]

## In a store

This is how the page indexing report in [Search Console](/glossary/google-search-console/) would read for a backpack store. It is a **fictitious example**: the URLs and statuses are made up to show how each one is interpreted; the status names and their meaning are those in Google's help.[^gsc-page-indexing][^gsc-url-inspection]

| URL (fictitious example)           | Status in Search Console                                             | What it means                                                                    | What to do                                                        |
| ---------------------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| `/urban-backpack-20l/`             | URL is on Google                                                     | The page is indexed and can show up in results, though that is not assured.      | Nothing, except making sure it stays that way.                    |
| `/urban-backpack-20l/?color=green` | Alternate page with proper [canonical tag](/glossary/canonical-url/) | It points correctly to its canonical, which is indexed.                          | Nothing: this is the right behavior.                              |
| `/backpacks/?sort=price`           | Duplicate without user-selected canonical                            | Google sees it as a copy of another page and nobody has told it which one rules. | Declare the canonical of sorted listings.                         |
| `/travel-backpack-40l/`            | Duplicate, Google chose different canonical than user                | You declare it canonical, but Google indexed another URL instead.                | Check which URL Google chose and why the signals do not match.    |
| `/backpack-2024-edition/`          | Soft 404                                                             | It shows a "not found" message without returning a 404 code.                     | Return a 404 or redirect to an equivalent product.                |
| `/laptop-sleeve-15/`               | Discovered - currently not indexed                                   | Google found it but has not crawled it yet.                                      | Wait and link to it from pages that are already crawled.          |
| `/search/?q=backpack`              | URL marked ‘noindex’                                                 | The store asks for it not to be indexed.                                         | Nothing, if it is intentional; internal results rarely add value. |

The row that usually costs the most money is "Crawled - currently not indexed," which does not appear in this example: Google visited the page and decided not to store it, although it may index it later.[^gsc-page-indexing] There is no switch for that; the page has to earn its place.


**Blocking a URL in `robots.txt` does not take it out of the index; sometimes it leaves it in.** If the page was already indexed or is linked from outside, the block stops the crawler from seeing the `noindex`, and the URL may keep showing up, now without a description.[^google-robots-intro][^google-noindex] To remove a discontinued product page or an internal results page, `noindex` first with crawling open; the block, if needed, after it has dropped out.


## How to check it

1. **Inspect a specific product page.** In Search Console, the URL Inspection tool tells you whether "URL is on Google" or "URL is not on Google," and shows you the user-declared canonical and the Google-selected one.[^gsc-url-inspection] If they do not match, that is your first job. Keep in mind that the "URL is on Google" status does not assure the page appears in results.[^gsc-url-inspection]
2. **Look at what Google sees, not what you see.** Click "Test Live URL" and then "View tested page": you get a screenshot of the rendered page, the HTML, the HTTP headers, the JavaScript console and the loaded resources.[^gsc-url-inspection] Look in that HTML for the product name, the price and the description. If they only appear when JavaScript runs, check that they also show up in the rendered HTML.[^google-js-seo]
3. **Do not read the live test as a verdict.** Compare it with the indexed version, which is the one from the last crawl. The live test does not check, among other things, whether the page is a duplicate of another or the quality guidelines.[^gsc-url-inspection]
4. **Review the page indexing report by reason**, not by total. Do not expect every URL on your site to be indexed: check that the canonicals of your important product and [category pages](/glossary/category-page/) are.[^gsc-page-indexing]
5. **Ask for it to come back once you have fixed it.** For a URL, "Request indexing," with a daily limit on requests and no guarantee the page gets into the index.[^gsc-url-inspection] For a whole reason, "Validate fix": it usually takes about two weeks and sometimes much longer.[^gsc-page-indexing]

## Related concept: crawling

Crawling is downloading the page; indexing is understanding it and deciding whether to store it.[^google-how-search-works] Search Console separates the two in its statuses: "Discovered - currently not indexed" means Google found the page but has not crawled it yet; "Crawled - currently not indexed," that it crawled it and did not index it.[^gsc-page-indexing] The first problem is one of access and priority; the second, of what the page offers compared with the others. The comparison [crawling vs indexing](/glossary/crawling-vs-indexing/) puts the two stages side by side.

To go further, the [crawling and indexing reading path](/learn/crawling-and-indexing/) orders the rest of the terms, and the guide to [what an AI crawler sees in your store](/guides/what-an-ai-crawler-sees/) goes down to the agents that are not Google.