How do caching and the CDN affect the bots that crawl your store?

Cache here is not the browser's, nor the "clear cache" button of an app: it is the layer between your server and whoever requests a page, and it often answers in the server's place. Almost every store has one in front, its platform's or its CDN's (content delivery network), and that changes what a crawler sees: it sees what the cache gives it, when the cache gives it, and sometimes it only sees the wall the CDN puts up against bots it does not recognize. For a store, that is where it is decided how long Google takes to see a new price and whether a product page stays in the index after a badly set security rule.

How it works

The standard is RFC 9111 (June 2022). It distinguishes two kinds of cache: the private one, dedicated to a single user and normally inside the browser, and the shared one, which stores responses to reuse them with more than one user and usually lives in an intermediary. A stored response is fresh as long as its age does not exceed its lifetime, and then it can be served without asking the origin server; after that it is stale.⁠^{[1]IETF, RFC Editor (June 2022). RFC 9111: HTTP Caching} A CDN is a shared cache at scale: Google describes it as an intermediary between your origin server and the user, whose main job has always been to store copies.⁠^{[2]Google Search Central Blog (December 24, 2024). Crawling December: CDNs and crawling}

Conditional requests

A crawler does not have to download a page again if it has not changed. Google's crawling infrastructure supports both forms in the standard: ETag with If-None-Match, and Last-Modified with If-Modified-Since. The crawler sends the value it saved last time; if it matches the current one, the server answers 304 (Not Modified) with no body, and saves generating the page and sending it.⁠^{[3]Google Search Central Blog (December 9, 2024). Crawling December: HTTP caching}

Google recommends ETag, because Last-Modified has date-format problems,⁠^{[4]Google Crawling Infrastructure. Overview of Google crawlers and fetchers} and suggests setting both if you can. If a single URL has several versions, for example for mobile and desktop, each should carry its own ETag. And it leaves it to you to decide which change deserves invalidating the cache: a significant change to the content, yes; updating the year in the footer, probably not.⁠^{[3]Google Search Central Blog (December 9, 2024). Crawling December: HTTP caching} It also advises setting a max-age in Cache-Control with the number of seconds you expect the content not to change, to help the crawler decide when to come back.⁠^{[4]Google Crawling Infrastructure. Overview of Google crawlers and fetchers}

The figure that gives context: according to Google, ten years ago 0.026% of its crawl requests could be resolved from cache, and in December 2024 only 0.017%.⁠^{[3]Google Search Central Blog (December 9, 2024). Crawling December: HTTP caching}

The CDN and the crawl rate

Google infers from the IP that a site is behind a CDN, and then raises the threshold at which it slows down crawling: it assumes the site can take more simultaneous requests. The catch is the first visit. The CDN's cache starts cold, so your server has to serve every URL at least once before the CDN helps; if you publish a lot of URLs at once, Google warns that crawling will be intense for a few days.⁠^{[2]Google Search Central Blog (December 24, 2024). Crawling December: CDNs and crawling} It is one of the pieces of the crawl budget.

If your pages change depending on a request header, the cache can only reuse a response when the headers named by Vary match those of the new request.⁠^{[1]IETF, RFC Editor (June 2022). RFC 9111: HTTP Caching} That is the point where caching meets content negotiation.

When the CDN stops the wrong bot

The same protection that stops an attack can stop Googlebot. Google separates two kinds of block, and they do not weigh the same:⁠^{[2]Google Search Central Blog (December 24, 2024). Crawling December: CDNs and crawling}

A verification challenge that lets people through does not let through the bots you want. Google, other search engines and other crawler operators publish their IP addresses so you can identify them and, if you see fit, exempt them from the CDN's firewall rules.⁠^{[2]Google Search Central Blog (December 24, 2024). Crawling December: CDNs and crawling} The rules for AI bots have their own entry, AI bot blocking.

In a store

Two visits from the same crawler to the product page of a fictional example store. The first gets the whole page, with its ETag and a max-age of one hour:

GET /products/stainless-moka-6 HTTP/1.1
Host: coffee-makers-example.com

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
ETag: "stainless-moka-6-v42"
Cache-Control: max-age=3600

The second, days later, sends that ETag. If the page has not changed, the response has no body:

GET /products/stainless-moka-6 HTTP/1.1
Host: coffee-makers-example.com
If-None-Match: "stainless-moka-6-v42"

HTTP/1.1 304 Not Modified
ETag: "stainless-moka-6-v42"

The rule that follows for a store: if you change the price, the availability or the page's structured data, the ETag has to change. An ETag that does not change with the content tells the crawler there is nothing new to read.

How to check it

1. Look at your product page's headers with curl -sI https://your-domain/products/your-product: look for ETag, Last-Modified and Cache-Control. With neither of the first two, a crawler cannot make conditional requests.⁠^{[3]Google Search Central Blog (December 9, 2024). Crawling December: HTTP caching}
2. Repeat the request with If-None-Match and the ETag the first one gave you. If the page has not changed, a 304 with no body should come back.⁠^{[3]Google Search Central Blog (December 9, 2024). Crawling December: HTTP caching}
3. Open Search Console's URL Inspection with a product page and look at the rendered screenshot. A challenge or a blank page points to the CDN.⁠^{[2]Google Search Central Blog (December 24, 2024). Crawling December: CDNs and crawling}
4. Review your CDN's bot rules against the IPs published by Google and the other crawler operators you care about, and do it from time to time: Google warns that an IP can end up on a blocklist without you knowing.⁠^{[2]Google Search Central Blog (December 24, 2024). Crawling December: CDNs and crawling}
5. Change a test price and request the page again. If the response stays the same, the cache is serving a fresh copy according to its max-age.⁠^{[1]IETF, RFC Editor (June 2022). RFC 9111: HTTP Caching}

Related concept: crawl budget

Crawl budget is how much a search engine crawls of your site; caching and the CDN decide how much each of those requests costs your server. A 304 with no body and a CDN answering from its copy make each visit cheaper, and a CDN also raises the rate Google allows itself.⁠^{[2]Google Search Central Blog (December 24, 2024). Crawling December: CDNs and crawling}⁠^{[3]Google Search Central Blog (December 9, 2024). Crawling December: HTTP caching} But caching does not make a page get indexed: it only decides what the crawler receives. The difference between the two is covered in crawling vs indexing.

To go further, Web Bot Auth explains how your CDN can recognize a bot by its signature and not just its IP, and the reading path for crawling and indexing orders the rest of the terms.