# Caching and CDNs for Bots: How They Affect Crawling

> How HTTP caching and the CDN decide what Googlebot and AI bots see in your store: ETag, 304, max-age and the blocks that drop pages from the index.

URL: https://clione.ai/glossary/cdn-caching-for-bots/
Lang: en

Cache here is not the browser's, nor the "clear cache" button of an app: it is the layer between your server and whoever requests a page, and it often answers in the server's place. Almost every store has one in front, its platform's or its CDN's (content delivery network), and that changes what a [crawler](/glossary/web-crawler/) sees: **it sees what the cache gives it, when the cache gives it**, and sometimes it only sees the wall the CDN puts up against bots it does not recognize. For a store, that is where it is decided how long Google takes to see a new price and whether a [product page](/glossary/product-page/) stays in the index after a badly set security rule.

## How it works

The standard is RFC 9111 (June 2022). It distinguishes two kinds of cache: the **private** one, dedicated to a single user and normally inside the browser, and the **shared** one, which stores responses to reuse them with more than one user and usually lives in an intermediary. A stored response is **fresh** as long as its age does not exceed its lifetime, and then it can be served without asking the origin server; after that it is **stale**.[^rfc9111] A CDN is a shared cache at scale: Google describes it as an intermediary between your origin server and the user, whose main job has always been to store copies.[^google-cdns]

### Conditional requests

A crawler does not have to download a page again if it has not changed. Google's crawling infrastructure supports both forms in the standard: `ETag` with `If-None-Match`, and `Last-Modified` with `If-Modified-Since`. The crawler sends the value it saved last time; if it matches the current one, the server answers **304 (_Not Modified_) with no body**, and saves generating the page and sending it.[^google-caching]

Google **recommends `ETag`**, because `Last-Modified` has date-format problems,[^google-crawlers] and suggests setting both if you can. If a single URL has several versions, for example for mobile and desktop, each should carry its own `ETag`. And it leaves it to you to decide which change deserves invalidating the cache: a significant change to the content, yes; updating the year in the footer, probably not.[^google-caching] It also advises setting a `max-age` in `Cache-Control` with the number of seconds you expect the content not to change, to help the crawler decide when to come back.[^google-crawlers]

The figure that gives context: according to Google, ten years ago 0.026% of its crawl requests could be resolved from cache, and in December 2024 **only 0.017%**.[^google-caching]

### The CDN and the crawl rate

Google infers from the IP that a site is behind a CDN, and then raises the threshold at which it slows down crawling: it assumes the site can take more simultaneous requests. The catch is the first visit. The CDN's cache starts cold, so **your server has to serve every URL at least once** before the CDN helps; if you publish a lot of URLs at once, Google warns that crawling will be intense for a few days.[^google-cdns] It is one of the pieces of the [crawl budget](/glossary/crawl-budget/).

If your pages change depending on a request header, the cache can only reuse a response when the headers named by `Vary` match those of the new request.[^rfc9111] That is the point where caching meets [content negotiation](/glossary/content-negotiation/).

## When the CDN stops the wrong bot

The same protection that stops an attack can stop [Googlebot](/glossary/googlebot/). Google separates two kinds of block, and they do not weigh the same:[^google-cdns]

| What the crawler receives       | What happens, according to Google                                                                                                                   |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| A 503 or a 429                  | It is the preferred way to signal a temporary block: it gives you time to react                                                                     |
| A timeout                       | It is treated as a permanent error: the affected URLs leave the index and the crawl rate may drop                                                   |
| An error page served with a 200 | If Google detects it as an error, it removes the URL; if not, pages with the same message may be dropped as duplicates, and recovering takes longer |
| An "are you human?" challenge   | The crawler only sees the challenge, not your page. Google recommends answering automated clients with a 503                                        |

**A verification challenge that lets people through does not let through the bots you want.** Google, other search engines and other crawler operators publish their IP addresses so you can identify them and, if you see fit, exempt them from the CDN's firewall rules.[^google-cdns] The rules for AI bots have their own entry, [AI bot blocking](/glossary/ai-bot-blocking/).

## In a store

Two visits from the same crawler to the product page of a **fictional example** store. The first gets the whole page, with its `ETag` and a `max-age` of one hour:

```http
GET /products/stainless-moka-6 HTTP/1.1
Host: coffee-makers-example.com

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
ETag: "stainless-moka-6-v42"
Cache-Control: max-age=3600
```

The second, days later, sends that `ETag`. If the page has not changed, the response has no body:

```http
GET /products/stainless-moka-6 HTTP/1.1
Host: coffee-makers-example.com
If-None-Match: "stainless-moka-6-v42"

HTTP/1.1 304 Not Modified
ETag: "stainless-moka-6-v42"
```

The rule that follows for a store: **if you change the price, the availability or the page's [structured data](/glossary/structured-data/), the `ETag` has to change**. An `ETag` that does not change with the content tells the crawler there is nothing new to read.


Taking the anti-bot challenge as fine because customers get through without trouble. The crawler is not a person and does not pretend to be one: when the challenge appears, **it is all it sees**. Google asks for a 503 for automated clients, so the page does not leave the index, and suggests looking at the rendered screenshot in URL Inspection in [Search Console](/glossary/google-search-console/): if you see the page, fine; if you see an error, a blank page or the challenge, it is time to talk to your CDN.[^google-cdns]


## How to check it

1. **Look at your product page's headers** with `curl -sI https://your-domain/products/your-product`: look for `ETag`, `Last-Modified` and `Cache-Control`. With neither of the first two, a crawler cannot make conditional requests.[^google-caching]
2. **Repeat the request with `If-None-Match`** and the `ETag` the first one gave you. If the page has not changed, a 304 with no body should come back.[^google-caching]
3. **Open Search Console's URL Inspection** with a product page and look at the rendered screenshot. A challenge or a blank page points to the CDN.[^google-cdns]
4. **Review your CDN's bot rules** against the IPs published by Google and the other crawler operators you care about, and do it from time to time: Google warns that an IP can end up on a blocklist without you knowing.[^google-cdns]
5. **Change a test price and request the page again.** If the response stays the same, the cache is serving a fresh copy according to its `max-age`.[^rfc9111]

## Related concept: crawl budget

Crawl budget is how much a search engine crawls of your site; caching and the CDN decide **how much each of those requests costs** your server. A 304 with no body and a CDN answering from its copy make each visit cheaper, and a CDN also raises the rate Google allows itself.[^google-cdns][^google-caching] But caching does not make a page get indexed: it only decides what the crawler receives. The difference between the two is covered in [crawling vs indexing](/glossary/crawling-vs-indexing/).

To go further, [Web Bot Auth](/glossary/web-bot-auth/) explains how your CDN can recognize a bot by its signature and not just its IP, and the [reading path](/learn/crawling-and-indexing/) for crawling and [indexing](/glossary/indexing/) orders the rest of the terms.