What is crawl budget?

Crawl budget is not money you pay Google or a number that shows up in any dashboard: it is a way of talking about how many of your site's URLs a crawler is willing to request and which ones it picks. It matters to a store for a concrete reason. A catalog with color, size and price filters produces far more URLs than products, and if the crawler spends its time on them, new product pages and price changes take longer to reach the search engine. It is just as useful to know the opposite: Google says that if your pages get crawled the same day they are published, you do not need to read its guide.⁠^{[1]Google Search Central. Optimize your crawl budget}

How it works

Google starts from an observation: the web is nearly infinite, and it has to put limits on the time and resources it spends on each site. It calls that allocation crawl budget, and defines it as the set of URLs that Google can and wants to crawl. For its infrastructure, a site is a hostname: www.store.com and blog.store.com have separate budgets.⁠^{[1]Google Search Central. Optimize your crawl budget}

The "can" is the crawl capacity limit, which Google also calls hostload: the total time your server spends with open connections for Google, counting how many at once and how long they last. Every site starts with the same conservative limit. If the server responds steadily and fast, the limit goes up; if it slows down or returns 5xx errors or throttling signals such as a 429, it goes down and Google crawls less. That limit is shared by all of Google's crawlers, so one crawler's demand reduces the capacity left for the others.⁠^{[1]Google Search Central. Optimize your crawl budget}

The "wants" is crawl demand. For Googlebot it depends on the size of the site, how often it changes, the quality of its pages and their relevance compared to other sites. Google names three factors you can influence: the inventory it perceives (without guidance from you, it tries to crawl all or nearly all the URLs it knows about, and it is the factor you can control most), the popularity of each URL and its staleness, because it wants to recrawl in time to pick up changes. There is one detail specific to commerce: Google Shopping has more demand for the products that are in your product feed in Google Merchant Center.⁠^{[1]Google Search Central. Optimize your crawl budget}

Two consequences that get forgotten. First: even if the capacity limit is not reached, if demand is low, Google crawls less. Second: crawling is not indexing. Every crawled page is evaluated and consolidated afterward, and not all of them end up in the index;⁠^{[1]Google Search Central. Optimize your crawl budget} that second phase is indexing. To get more budget, Google gives only two routes: more server resources, if the problem is capacity, and more quality for the Google product you are aiming at, which in Search includes popularity, value to the user, unique content and the ability to serve it.⁠^{[1]Google Search Central. Optimize your crawl budget}

Who is all this for? Google writes it for three kinds of site: those with over a million unique pages that change about once a week, those with over 10,000 that change daily and those with a large share of their URLs in the "Discovered - currently not indexed" status in Search Console. And it warns that the numbers are a rough estimate, not exact thresholds.⁠^{[1]Google Search Central. Optimize your crawl budget}

In a store

A clothing store with 2,000 products and 40 collections. Each collection can be filtered by six colors, five sizes and four sort orders, and each combination generates its own URL with parameters. This is a fictitious example, with round figures to see the proportion:

The useful catalog is 2,040 URLs; the one the crawler can find is almost 16,000. Google describes the two harms caused by faceted navigation: overcrawling, because every filter URL looks new and the crawler does not know whether it is useful until it requests it, and slower discovery, because the time spent on useless URLs leaves less for the new ones.⁠^{[2]Google Search Central. Managing crawling of faceted navigation URLs} The same goes for duplicate content: Google prefers that Googlebot spend its time on new or updated pages and not on duplicate versions of the same content.⁠^{[3]Google Search Central. How to specify a canonical URL with rel="canonical" and other methods}

The ways out that Google proposes for filters you do not want crawled: block them in robots.txt, use fragments (#color=blue) instead of parameters, or mark their links with rel="nofollow". A canonical URL pointing to the unfiltered collection can also be used, but Google considers it generally less effective in the long run.⁠^{[2]Google Search Central. Managing crawling of faceted navigation URLs} None of these figures says this store has a problem: with 2,000 products, it is far below the audience for the guide. What would say so is the Crawl Stats report.

How to check it

1. First see whether it is your problem. Publish a product and check in Search Console's URL Inspection when Google crawled it. If it is the same day, Google says it is enough to keep the sitemap up to date and regularly review the page indexing report.⁠^{[1]Google Search Central. Optimize your crawl budget}
2. Open the Crawl Stats report from the property settings. It gives total requests, download size, average response time and host status, with breakdowns by response code, file type, purpose (discovery or refresh) and Googlebot type. It only works on root-level properties, and Google presents it as a report for advanced users that a site with fewer than 1,000 pages normally does not need.⁠^{[4]Search Console Help. Crawl Stats report}
3. Read where the crawling goes. Lots of 404 responses, redirects or pages that return a "not found" with a 200 (soft 404s) are wasted crawling. Google asks you to return a 404 or a 410 for products removed for good, eliminate soft 404s and avoid long redirect chains.⁠^{[1]Google Search Central. Optimize your crawl budget}
4. Sort out your URL inventory. Consolidate duplicates, decide which filters are not crawled and keep the XML sitemap with the <lastmod> tag when content changes.⁠^{[1]Google Search Central. Optimize your crawl budget}⁠^{[2]Google Search Central. Managing crawling of faceted navigation URLs}
5. Watch server speed. If responses lengthen or 5xx or 429 errors appear, the capacity limit goes down. Answering unchanged pages with a 304 (Not Modified) lets Google reuse its copy and saves your server resources.⁠^{[1]Google Search Central. Optimize your crawl budget}

Related concept: indexing

Crawl budget decides which URLs get requested; indexing, which of those get stored. That is why a generous budget does not fix a page that Google crawls and does not index: the "Crawled - currently not indexed" status speaks to what the page offers, not to how much is crawled. The one that does point to crawling is "Discovered - currently not indexed", the third profile Google aims its guide at.⁠^{[1]Google Search Central. Optimize your crawl budget} The entry on crawling vs indexing puts the two phases side by side, and the one on IndexNow explains how to tell Bing and other search engines about a change without waiting for them to discover it.

To go further, the crawling and indexing reading path orders the rest of the terms.