# What Is Googlebot and How to Tell If a Visit Is Real

> What Googlebot is, Google Search's crawler, what it reads on your product pages, what happens if you block it and how to verify a visit is really from Google.

URL: https://clione.ai/glossary/googlebot/
Lang: en

Googlebot is Google Search's [crawler](/glossary/web-crawler/): the program that requests your pages so Google can index and show them. It is not a tool you install or a Google account, and it is not Google's only agent, but it is the one that crawls your product pages for Search. For a store, knowing it serves three purposes: knowing **what it really reads on each [product page](/glossary/product-page/)**, not shutting the door on it by mistake and telling it apart from the many bots that use its name without being it.

## How it works

Google calls two crawlers Googlebot: Googlebot Smartphone, which simulates a person on a mobile device, and Googlebot Desktop, which simulates one on a desktop computer. Because on most sites Google mainly indexes the mobile version of the content, **most requests are made by the mobile crawler**. Both obey the same name in [robots.txt](/glossary/robots-txt/), so you cannot allow one and block the other from there.[^google-googlebot]

Google sorts its agents into three groups. Common crawlers, like Googlebot, **always respect robots.txt rules in their automatic crawls**. Special-case crawlers work for specific products under an agreement between the site and the product, and user-triggered fetchers make a request when someone asks for it from a tool.[^google-crawlers-overview] What you allow or block for Googlebot affects Google Search, with Discover and all its features, and also Google Images, Google Video and Google News.[^google-common-crawlers]

That includes AI. Google explains that AI is integrated into Search and that therefore **robots.txt directives for Googlebot are the control** for managing how a site is crawled for it: there is no separate crawler for [AI Overviews](/glossary/ai-overviews/) or for [AI Mode](/glossary/google-ai-mode/).[^google-ai-features] The name that does exist separately is [Google-Extended](/glossary/google-extended/), and it is not a crawler: it decides whether what Google crawls can be used to train [Gemini](/glossary/gemini/) models, and Google says it does not affect your site's inclusion in Search.[^google-common-crawlers]

Googlebot asks for little and with care. Google says it should not access a site **more than once every few seconds on average**.[^google-googlebot] The practical limit on how much gets crawled on a site is what is known as [crawl budget](/glossary/crawl-budget/).

And it does not read everything. Googlebot crawls **the first 2 MB of each supported file type**, and the first 64 MB of a PDF. The limit applies to uncompressed data. When it reaches the cutoff, it stops downloading and only the part it already has moves on to [indexing](/glossary/indexing/). Each resource the HTML links to, such as CSS or JavaScript, is downloaded separately, with the same limit.[^google-googlebot] A product page whose HTML is bloated with inline scripts can push whatever comes at the end beyond those 2 MB.

The request arrives with one of these two [user agents](/glossary/user-agent/), according to the October 2026 documentation. `W.X.Y.Z` is the Chrome version, which changes:[^google-common-crawlers]

```text
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
```

A request saying that proves nothing. Google warns that **other crawlers often fake Googlebot's [user agent](/glossary/user-agent/)**, and that the best way to verify is a reverse DNS lookup on the source IP or comparing that IP with Googlebot's ranges.[^google-googlebot] Common crawlers come from the ranges published in `common-crawlers.json`, and their hostname follows the pattern `crawl-***-***-***-***.googlebot.com` or `geo-crawl-***-***-***-***.geo.googlebot.com`.[^google-common-crawlers] The version of the file we read on October 6, 2026 was dated that same day, with 170 IPv4 prefixes and 147 IPv6.[^google-ips] It is regenerated often, so it gets checked at the time you verify.

## In a store

A photography gear store sees in its logs a thousand requests in an hour that claim to be Googlebot, all to filtered collection URLs. It is a **fictional example**, with a simplified line and an IP reserved for documentation:

```text
203.0.113.77 [06/Oct/2026:04:10:12 +0000] "GET /collections/tripods?filter.p.vendor=X&sort_by=price HTTP/1.1" 200 "...compatible; Googlebot/2.1; +http://www.google.com/bot.html..."
```

Before touching robots.txt, the store verifies the IP with Google's manual method, in four steps:[^google-verify]

1. Run a reverse DNS lookup on the IP with the `host` command.
2. Check that the name it returns ends in `googlebot.com`, `google.com` or `googleusercontent.com`.
3. Run a forward DNS lookup with `host` on that name.
4. Check that it returns the same IP as in the log.

If the name does not end in one of those domains, or the forward DNS returns a different IP, the visit is not from Google, and **blocking that traffic does not affect Search**. If the IP is Google's, the problem is a different one: the filtered collections are open to crawling and Googlebot is walking through them. There the fix is a robots.txt rule for those URLs, not a block on Googlebot.


**Blocking Googlebot to stop Google's AI does not take you out of the AI alone.** The robots.txt directives for Googlebot are the control for all of Search, because AI is integrated into it.[^google-ai-features] Shutting the door on it to get out of AI Overviews keeps it from crawling your product pages for everything else: the regular results, Discover, Google Images and Google News.[^google-common-crawlers] To limit what is shown in the AI features, Google points to `nosnippet`, `data-nosnippet`, `max-snippet` and [`noindex`](/glossary/noindex/).[^google-ai-features]


## How to check it

1. **Open `yourdomain.com/robots.txt` and look for a group that names Googlebot.** If there is none, the asterisk group applies. See what that group and the general one say about the path of your product pages.
2. **Verify the visits that claim to be Googlebot** with reverse and forward DNS, or against `common-crawlers.json`.[^google-verify]
3. **Look at the uncompressed weight of your product pages' HTML.** If it gets close to 2 MB, whatever comes at the end may be left out.[^google-googlebot]
4. **Use the URL Inspection tool in [Search Console](/glossary/google-search-console/)** to see the HTML Googlebot received when it crawled a page. Google recommends it for checking that the implementation is correct.[^google-ai-features]
5. **Check the firewall or the CDN.** A bot rule that stops Google's IPs blocks crawling even if robots.txt allows it.

## Related concept: Google-Extended

Googlebot visits your store; Google-Extended never does. Google-Extended has no user agent of its own: crawling is done by Google's usual agents, and the name in robots.txt only decides whether what was crawled can be used to train Gemini models and for [grounding](/glossary/grounding/) in the Gemini apps and in Vertex AI.[^google-common-crawlers] **Blocking Googlebot changes what Google can read; blocking Google-Extended only changes whether it can use it to train and feed Gemini.**

To go further, the entry on [crawling vs indexing](/glossary/crawling-vs-indexing/) separates what Googlebot does from what Google decides afterward, and the [reading path](/learn/crawling-and-indexing/) for this topic orders the rest of the terms.