# What Is Google-Extended and What Does It Block?

> What Google-Extended is, the robots.txt control for Gemini training, why it does not take you out of AI Overviews or AI Mode and what to use for that.

URL: https://clione.ai/glossary/google-extended/
Lang: en

Google-Extended is not a robot that will show up in your logs, or an extended version of Google: it is a word you write in [robots.txt](/glossary/robots-txt/) to answer a single question, whether what Google crawls on your site can be used to train and feed its [Gemini](/glossary/gemini/) models. For a store it matters because it usually shows up in the same bundle as [GPTBot](/glossary/gptbot/) or [ClaudeBot](/glossary/claudebot/), and it does not work like them. **Blocking it changes neither what Google crawls nor what it shows in Search**, its AI answers included.

## How it works

Google introduced Google-Extended on September 28, 2023 as a new control for web publishers to decide whether their sites help improve Bard and Vertex AI generative APIs, including future generations of the models that power those products.[^google-publisher-controls] Google's current documentation no longer mentions Bard, and describes it this way: a standalone name publishers can use to decide whether the content Google crawls from their sites can be used to **train future generations of the Gemini models** that power Gemini apps and the Vertex AI API for Gemini, and for [grounding](/glossary/grounding/) in those products, that is, to base their answers on retrieved pages.[^google-common-crawlers]

What makes it different from any other robots.txt name is that **it has no visits of its own**. Google says Google-Extended has no separate [user agent](/glossary/user-agent/) string: crawling is done with Google's usual agents, and the robots.txt name only serves a control function.[^google-common-crawlers] [Googlebot](/glossary/googlebot/) downloads your page as always; what changes, if you block Google-Extended, is what it can be used for afterward.

Google also clarifies what it does not do: it **does not affect a site's inclusion in Google Search and is not used as a ranking signal**.[^google-common-crawlers] And the [Search Console](/glossary/google-search-console/) help adds one use: to limit training of the models that generate the answers of Search's generative AI features, it also points to Google-Extended.[^gsc-ai-control] Limiting training is not leaving those answers, and Google has other controls for that.

Search's AI features, [AI Overviews](/glossary/ai-overviews/) and [AI Mode](/glossary/google-ai-mode/), follow Search's rules. Google explains that AI is integrated into Search and that therefore **robots.txt directives for Googlebot are the control** for managing how a site is crawled for it.[^google-ai-features] To appear as a supporting link in those features, a page has to be indexed and eligible to be shown with a snippet, and what is shown is limited with `nosnippet`, `data-nosnippet`, `max-snippet` or [`noindex`](/glossary/noindex/). To limit training and grounding in some of Google's other systems, the same page points to Google-Extended.[^google-ai-features]

Since August 31, 2026 there is one more control, in Search Console, available to all sites worldwide: the [Search generative AI control](/glossary/search-console-generative-ai-control/), in Settings. With the opt-out option, **the site's links and content stop being shown in AI Overviews, in AI Mode and in Discover's generative AI features**. Google warns that it takes a few days to apply, that it is not a ranking signal and that the content can still be used to make Search work in general.[^gsc-ai-control]

Because Google-Extended is a robots.txt name, the standard's rules apply to it the same way: a group that names it is matched case-insensitively, and if there is none the `User-agent: *` group is followed.[^rfc9309] Google gives this example, which lets one section of a site through and closes the rest:[^google-common-crawlers]

```text
user-agent: Google-Extended
allow: /archive/1Q84
disallow: /archive/
```

## In a store

An antiquarian bookstore wants three things: to stay in Search, to keep appearing in AI Overviews and for its catalog not to be used to train Gemini. It is a **fictional example**. What it has to touch, control by control:

| What it wants                                                                            | Control                                      | Where it is changed         |
| ---------------------------------------------------------------------------------------- | -------------------------------------------- | --------------------------- |
| Google to crawl and index the product pages                                              | robots.txt rules for Googlebot               | robots.txt                  |
| Its content not to train Gemini models or ground answers in the Gemini apps or Vertex AI | Google-Extended                              | robots.txt                  |
| To appear or not in AI Overviews, AI Mode and Discover's AI features                     | Search generative AI control                 | Search Console, Settings    |
| To limit the text shown from a page                                                      | `nosnippet`, `data-nosnippet`, `max-snippet` | The page's HTML             |
| For a page not to appear in Search                                                       | `noindex`                                    | The HTML or the HTTP header |

The sources for each row are Google's [crawler](/glossary/web-crawler/) documentation, its page on AI features and the Search Console help.[^google-common-crawlers][^google-ai-features][^gsc-ai-control] Its robots.txt ends up like this:

```text
User-agent: *
Disallow: /cart
Disallow: /checkout

User-agent: Google-Extended
Disallow: /
```

Googlebot follows the asterisk group and crawls the product pages as before. Google-Extended has its own group and is closed off entirely. Nothing will change in the server logs: **you will never see a request from Google-Extended**, because it makes none.[^google-common-crawlers]


**Blocking Google-Extended does not take you out of AI Overviews or AI Mode.** Google points to Google-Extended to limit training and grounding in some of its other systems, and for limiting what is shown in Search it points to the snippet controls and `noindex`.[^google-ai-features] To exclude the site from Search's generative AI features there is the Search Console control.[^gsc-ai-control] And the other way around: blocking Googlebot to stop Google's AI keeps it from crawling your pages for all of Search, not just its AI features.[^google-ai-features]


## How to check it

1. **Open `yourdomain.com/robots.txt` and look for a group that names Google-Extended.** If there is none, the asterisk group applies.[^rfc9309]
2. **Check that the Googlebot group, or the asterisk one, still lets your product pages through.** It is the one that counts for Search and its AI features.[^google-ai-features]
3. **Don't look for Google-Extended in the logs.** It has no user agent of its own; the visits you will see are from Google's usual agents.[^google-common-crawlers]
4. **Open Search Console, Settings, and look at the Search generative AI control.** By default it includes the site; if someone chose to exclude it, that is where you see it.[^gsc-ai-control]
5. **Check whether your product pages carry `nosnippet` or a short `max-snippet`.** They cut down what AI Overviews and AI Mode can use, with Google-Extended or without it.[^google-ai-features]

## Related concept: GPTBot

Both are named in robots.txt to decide about training, which is why they show up together in lists. But GPTBot is a real [crawler](/glossary/web-crawler/): OpenAI uses it to crawl content that may be used to train its models, publishes its user agent and its IPs, and separates it from [OAI-SearchBot](/glossary/oai-searchbot/), the agent for its search.[^openai-bots] Google-Extended crawls nothing and has no IP to verify: **it is a permission over what Googlebot already crawls**, and Google separates it from Search.[^google-common-crawlers] Blocking GPTBot closes a door to a visitor; blocking Google-Extended changes the use made of what comes in through another door.

To go further, the [Googlebot](/glossary/googlebot/) entry explains the crawler that does visit your store, the guide to [what an AI crawler sees on your store](/guides/what-an-ai-crawler-sees/) goes agent by agent through what each one reads, and the [reading path](/learn/crawling-and-indexing/) for this topic orders the rest of the terms.