What is Googlebot, Google's crawler?

Googlebot is Google Search's crawler: the program that requests your pages so Google can index and show them. It is not a tool you install or a Google account, and it is not Google's only agent, but it is the one that crawls your product pages for Search. For a store, knowing it serves three purposes: knowing what it really reads on each product page, not shutting the door on it by mistake and telling it apart from the many bots that use its name without being it.

How it works

Google calls two crawlers Googlebot: Googlebot Smartphone, which simulates a person on a mobile device, and Googlebot Desktop, which simulates one on a desktop computer. Because on most sites Google mainly indexes the mobile version of the content, most requests are made by the mobile crawler. Both obey the same name in robots.txt, so you cannot allow one and block the other from there.⁠^{[1]Google Search Central. What is Googlebot}

Google sorts its agents into three groups. Common crawlers, like Googlebot, always respect robots.txt rules in their automatic crawls. Special-case crawlers work for specific products under an agreement between the site and the product, and user-triggered fetchers make a request when someone asks for it from a tool.⁠^{[2]Google Crawling Infrastructure. Google crawler (user agent) overview} What you allow or block for Googlebot affects Google Search, with Discover and all its features, and also Google Images, Google Video and Google News.⁠^{[3]Google Crawling Infrastructure. Google's common crawlers}

That includes AI. Google explains that AI is integrated into Search and that therefore robots.txt directives for Googlebot are the control for managing how a site is crawled for it: there is no separate crawler for AI Overviews or for AI Mode.⁠^{[4]Google Search Central. AI features and your website} The name that does exist separately is Google-Extended, and it is not a crawler: it decides whether what Google crawls can be used to train Gemini models, and Google says it does not affect your site's inclusion in Search.⁠^{[3]Google Crawling Infrastructure. Google's common crawlers}

Googlebot asks for little and with care. Google says it should not access a site more than once every few seconds on average.⁠^{[1]Google Search Central. What is Googlebot} The practical limit on how much gets crawled on a site is what is known as crawl budget.

And it does not read everything. Googlebot crawls the first 2 MB of each supported file type, and the first 64 MB of a PDF. The limit applies to uncompressed data. When it reaches the cutoff, it stops downloading and only the part it already has moves on to indexing. Each resource the HTML links to, such as CSS or JavaScript, is downloaded separately, with the same limit.⁠^{[1]Google Search Central. What is Googlebot} A product page whose HTML is bloated with inline scripts can push whatever comes at the end beyond those 2 MB.

The request arrives with one of these two user agents, according to the October 2026 documentation. W.X.Y.Z is the Chrome version, which changes:⁠^{[3]Google Crawling Infrastructure. Google's common crawlers}

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

A request saying that proves nothing. Google warns that other crawlers often fake Googlebot's user agent, and that the best way to verify is a reverse DNS lookup on the source IP or comparing that IP with Googlebot's ranges.⁠^{[1]Google Search Central. What is Googlebot} Common crawlers come from the ranges published in common-crawlers.json, and their hostname follows the pattern crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.⁠^{[3]Google Crawling Infrastructure. Google's common crawlers} The version of the file we read on October 6, 2026 was dated that same day, with 170 IPv4 prefixes and 147 IPv6.⁠^{[5]Google. common-crawlers.json (IP ranges of Google's common crawlers)} It is regenerated often, so it gets checked at the time you verify.

In a store

A photography gear store sees in its logs a thousand requests in an hour that claim to be Googlebot, all to filtered collection URLs. It is a fictional example, with a simplified line and an IP reserved for documentation:

203.0.113.77 [06/Oct/2026:04:10:12 +0000] "GET /collections/tripods?filter.p.vendor=X&sort_by=price HTTP/1.1" 200 "...compatible; Googlebot/2.1; +http://www.google.com/bot.html..."

Before touching robots.txt, the store verifies the IP with Google's manual method, in four steps:⁠^{[6]Google Crawling Infrastructure. Verify requests from Google crawlers and fetchers}

1. Run a reverse DNS lookup on the IP with the host command.
2. Check that the name it returns ends in googlebot.com, google.com or googleusercontent.com.
3. Run a forward DNS lookup with host on that name.
4. Check that it returns the same IP as in the log.

If the name does not end in one of those domains, or the forward DNS returns a different IP, the visit is not from Google, and blocking that traffic does not affect Search. If the IP is Google's, the problem is a different one: the filtered collections are open to crawling and Googlebot is walking through them. There the fix is a robots.txt rule for those URLs, not a block on Googlebot.

How to check it

1. Open yourdomain.com/robots.txt and look for a group that names Googlebot. If there is none, the asterisk group applies. See what that group and the general one say about the path of your product pages.
2. Verify the visits that claim to be Googlebot with reverse and forward DNS, or against common-crawlers.json.⁠^{[6]Google Crawling Infrastructure. Verify requests from Google crawlers and fetchers}
3. Look at the uncompressed weight of your product pages' HTML. If it gets close to 2 MB, whatever comes at the end may be left out.⁠^{[1]Google Search Central. What is Googlebot}
4. Use the URL Inspection tool in Search Console to see the HTML Googlebot received when it crawled a page. Google recommends it for checking that the implementation is correct.⁠^{[4]Google Search Central. AI features and your website}
5. Check the firewall or the CDN. A bot rule that stops Google's IPs blocks crawling even if robots.txt allows it.

Related concept: Google-Extended

Googlebot visits your store; Google-Extended never does. Google-Extended has no user agent of its own: crawling is done by Google's usual agents, and the name in robots.txt only decides whether what was crawled can be used to train Gemini models and for grounding in the Gemini apps and in Vertex AI.⁠^{[3]Google Crawling Infrastructure. Google's common crawlers} Blocking Googlebot changes what Google can read; blocking Google-Extended only changes whether it can use it to train and feed Gemini.

To go further, the entry on crawling vs indexing separates what Googlebot does from what Google decides afterward, and the reading path for this topic orders the rest of the terms.