# What Is Crawl Budget in SEO? When It Matters for Stores > What crawl budget is, when an online store should care, what wastes it (filters, duplicates, noindex) and how to measure it. URL: https://clione.ai/glossary/crawl-budget/ Lang: en Crawl budget is not money you pay Google or a number that shows up in any dashboard: it is a way of talking about **how many of your site's URLs a [crawler](/glossary/web-crawler/) is willing to request** and which ones it picks. It matters to a store for a concrete reason. A catalog with color, size and price filters produces far more URLs than products, and if the crawler spends its time on them, new [product pages](/glossary/product-page/) and price changes take longer to reach the search engine. It is just as useful to know the opposite: Google says that **if your pages get crawled the same day they are published, you do not need to read its guide**.[^google-crawl-budget] ## How it works Google starts from an observation: the web is nearly infinite, and it has to put limits on the time and resources it spends on each site. It calls that allocation crawl budget, and defines it as **the set of URLs that Google can and wants to crawl**. For its infrastructure, a site is a hostname: `www.store.com` and `blog.store.com` have separate budgets.[^google-crawl-budget] The "can" is the **crawl capacity limit**, which Google also calls _hostload_: the total time your server spends with open connections for Google, counting how many at once and how long they last. Every site starts with the same conservative limit. If the server responds steadily and fast, the limit goes up; if it slows down or returns 5xx errors or throttling signals such as a 429, it goes down and Google crawls less. That limit is shared by all of Google's crawlers, so one crawler's demand reduces the capacity left for the others.[^google-crawl-budget] The "wants" is **crawl demand**. For [Googlebot](/glossary/googlebot/) it depends on the size of the site, how often it changes, the quality of its pages and their relevance compared to other sites. Google names three factors you can influence: the inventory it perceives (without guidance from you, it tries to crawl all or nearly all the URLs it knows about, and it is the factor you can control most), the popularity of each URL and its staleness, because it wants to recrawl in time to pick up changes. There is one detail specific to commerce: Google Shopping has more demand for the products that are in your [product feed](/glossary/product-feed/) in [Google Merchant Center](/glossary/google-merchant-center/).[^google-crawl-budget] Two consequences that get forgotten. First: even if the capacity limit is not reached, **if demand is low, Google crawls less**. Second: crawling is not [indexing](/glossary/indexing/). Every crawled page is evaluated and consolidated afterward, and not all of them end up in the index;[^google-crawl-budget] that second phase is [indexing](/glossary/indexing/). To get more budget, Google gives only two routes: more server resources, if the problem is capacity, and more quality for the Google product you are aiming at, which in Search includes popularity, value to the user, unique content and the ability to serve it.[^google-crawl-budget] Who is all this for? Google writes it for three kinds of site: those with over a million unique pages that change about once a week, those with over 10,000 that change daily and those with a large share of their URLs in the "Discovered - currently not indexed" status in [Search Console](/glossary/google-search-console/). And it warns that **the numbers are a rough estimate, not exact thresholds**.[^google-crawl-budget] ## In a store A clothing store with 2,000 products and 40 collections. Each collection can be filtered by six colors, five sizes and four sort orders, and each combination generates its own URL with parameters. This is a **fictitious example**, with round figures to see the proportion: | URL type (fictitious example) | How many | Do you want Google to crawl it? | | ----------------------------------------------------------------------------- | -------- | -------------------------------------- | | [Product pages](/glossary/product-page/) | 2,000 | Yes | | Unfiltered collections | 40 | Yes | | Collections with color, size and sort (40 × 6 × 5 × 4) | 4,800 | Almost never | | Product pages with `?variant=` in the URL | 9,000 | No: they point to their canonical page | The useful catalog is 2,040 URLs; the one the crawler can find is almost 16,000. Google describes the two harms caused by [faceted navigation](/glossary/faceted-navigation/): overcrawling, because every filter URL looks new and the crawler does not know whether it is useful until it requests it, and **slower discovery**, because the time spent on useless URLs leaves less for the new ones.[^google-faceted] The same goes for [duplicate content](/glossary/duplicate-content/): Google prefers that Googlebot spend its time on new or updated pages and not on duplicate versions of the same content.[^google-canonical] The ways out that Google proposes for filters you do not want crawled: block them in [`robots.txt`](/glossary/robots-txt/), use fragments (`#color=blue`) instead of parameters, or mark their links with `rel="nofollow"`. A [canonical URL](/glossary/canonical-url/) pointing to the unfiltered collection can also be used, but Google considers it **generally less effective in the long run**.[^google-faceted] None of these figures says this store has a problem: with 2,000 products, it is far below the audience for the guide. What would say so is the Crawl Stats report. **Marking the surplus pages with [`noindex`](/glossary/noindex/) does not save crawling.** Google still requests them and only drops them when it reads the rule, so the crawl time is spent all the same. For what you do not want crawled, Google recommends `robots.txt`. And blocking some URLs does not redistribute their budget among the rest: **Google does not move it to other pages unless it is already hitting the site's capacity limit**.[^google-crawl-budget] ## How to check it 1. **First see whether it is your problem.** Publish a product and check in Search Console's URL Inspection when Google crawled it. If it is the same day, Google says it is enough to keep the sitemap up to date and regularly review the page indexing report.[^google-crawl-budget] 2. **Open the Crawl Stats report** from the property settings. It gives total requests, download size, average response time and host status, with breakdowns by response code, file type, purpose (discovery or refresh) and Googlebot type. It only works on root-level properties, and Google presents it as a report for advanced users that a site with fewer than 1,000 pages normally does not need.[^gsc-crawl-stats] 3. **Read where the crawling goes.** Lots of 404 responses, redirects or pages that return a "not found" with a 200 (_soft 404s_) are wasted crawling. Google asks you to return a 404 or a 410 for products removed for good, eliminate soft 404s and avoid long redirect chains.[^google-crawl-budget] 4. **Sort out your URL inventory.** Consolidate duplicates, decide which filters are not crawled and keep the [XML sitemap](/glossary/xml-sitemap/) with the `` tag when content changes.[^google-crawl-budget][^google-faceted] 5. **Watch server speed.** If responses lengthen or 5xx or 429 errors appear, the capacity limit goes down. Answering unchanged pages with a 304 (_Not Modified_) lets Google reuse its copy and saves your server resources.[^google-crawl-budget] ## Related concept: indexing Crawl budget decides which URLs get requested; indexing, which of those get stored. That is why a generous budget does not fix a page that Google crawls and does not index: the "Crawled - currently not indexed" status speaks to what the page offers, not to how much is crawled. The one that does point to crawling is "Discovered - currently not indexed", the third profile Google aims its guide at.[^google-crawl-budget] The entry on [crawling vs indexing](/glossary/crawling-vs-indexing/) puts the two phases side by side, and the one on [IndexNow](/glossary/indexnow/) explains how to tell Bing and other search engines about a change without waiting for them to discover it. To go further, the [crawling and indexing reading path](/learn/crawling-and-indexing/) orders the rest of the terms.