What is Google-Extended and what does it actually block?

Google-Extended is not a robot that will show up in your logs, or an extended version of Google: it is a word you write in robots.txt to answer a single question, whether what Google crawls on your site can be used to train and feed its Gemini models. For a store it matters because it usually shows up in the same bundle as GPTBot or ClaudeBot, and it does not work like them. Blocking it changes neither what Google crawls nor what it shows in Search, its AI answers included.

How it works

Google introduced Google-Extended on September 28, 2023 as a new control for web publishers to decide whether their sites help improve Bard and Vertex AI generative APIs, including future generations of the models that power those products.⁠^{[1]Google (Danielle Romain, VP, Trust). An update on web publisher controls} Google's current documentation no longer mentions Bard, and describes it this way: a standalone name publishers can use to decide whether the content Google crawls from their sites can be used to train future generations of the Gemini models that power Gemini apps and the Vertex AI API for Gemini, and for grounding in those products, that is, to base their answers on retrieved pages.⁠^{[2]Google Crawling Infrastructure. Google's common crawlers}

What makes it different from any other robots.txt name is that it has no visits of its own. Google says Google-Extended has no separate user agent string: crawling is done with Google's usual agents, and the robots.txt name only serves a control function.⁠^{[2]Google Crawling Infrastructure. Google's common crawlers} Googlebot downloads your page as always; what changes, if you block Google-Extended, is what it can be used for afterward.

Google also clarifies what it does not do: it does not affect a site's inclusion in Google Search and is not used as a ranking signal.⁠^{[2]Google Crawling Infrastructure. Google's common crawlers} And the Search Console help adds one use: to limit training of the models that generate the answers of Search's generative AI features, it also points to Google-Extended.⁠^{[3]Search Console Help. Search generative AI control} Limiting training is not leaving those answers, and Google has other controls for that.

Search's AI features, AI Overviews and AI Mode, follow Search's rules. Google explains that AI is integrated into Search and that therefore robots.txt directives for Googlebot are the control for managing how a site is crawled for it.⁠^{[4]Google Search Central. AI features and your website} To appear as a supporting link in those features, a page has to be indexed and eligible to be shown with a snippet, and what is shown is limited with nosnippet, data-nosnippet, max-snippet or noindex. To limit training and grounding in some of Google's other systems, the same page points to Google-Extended.⁠^{[4]Google Search Central. AI features and your website}

Since August 31, 2026 there is one more control, in Search Console, available to all sites worldwide: the Search generative AI control, in Settings. With the opt-out option, the site's links and content stop being shown in AI Overviews, in AI Mode and in Discover's generative AI features. Google warns that it takes a few days to apply, that it is not a ranking signal and that the content can still be used to make Search work in general.⁠^{[3]Search Console Help. Search generative AI control}

Because Google-Extended is a robots.txt name, the standard's rules apply to it the same way: a group that names it is matched case-insensitively, and if there is none the User-agent: * group is followed.⁠^{[5]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} Google gives this example, which lets one section of a site through and closes the rest:⁠^{[2]Google Crawling Infrastructure. Google's common crawlers}

user-agent: Google-Extended
allow: /archive/1Q84
disallow: /archive/

In a store

An antiquarian bookstore wants three things: to stay in Search, to keep appearing in AI Overviews and for its catalog not to be used to train Gemini. It is a fictional example. What it has to touch, control by control:

The sources for each row are Google's crawler documentation, its page on AI features and the Search Console help.⁠^{[2]Google Crawling Infrastructure. Google's common crawlers}⁠^{[4]Google Search Central. AI features and your website}⁠^{[3]Search Console Help. Search generative AI control} Its robots.txt ends up like this:

User-agent: *
Disallow: /cart
Disallow: /checkout

User-agent: Google-Extended
Disallow: /

Googlebot follows the asterisk group and crawls the product pages as before. Google-Extended has its own group and is closed off entirely. Nothing will change in the server logs: you will never see a request from Google-Extended, because it makes none.⁠^{[2]Google Crawling Infrastructure. Google's common crawlers}

How to check it

1. Open yourdomain.com/robots.txt and look for a group that names Google-Extended. If there is none, the asterisk group applies.⁠^{[5]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}
2. Check that the Googlebot group, or the asterisk one, still lets your product pages through. It is the one that counts for Search and its AI features.⁠^{[4]Google Search Central. AI features and your website}
3. Don't look for Google-Extended in the logs. It has no user agent of its own; the visits you will see are from Google's usual agents.⁠^{[2]Google Crawling Infrastructure. Google's common crawlers}
4. Open Search Console, Settings, and look at the Search generative AI control. By default it includes the site; if someone chose to exclude it, that is where you see it.⁠^{[3]Search Console Help. Search generative AI control}
5. Check whether your product pages carry nosnippet or a short max-snippet. They cut down what AI Overviews and AI Mode can use, with Google-Extended or without it.⁠^{[4]Google Search Central. AI features and your website}

Related concept: GPTBot

Both are named in robots.txt to decide about training, which is why they show up together in lists. But GPTBot is a real crawler: OpenAI uses it to crawl content that may be used to train its models, publishes its user agent and its IPs, and separates it from OAI-SearchBot, the agent for its search.⁠^{[6]OpenAI. Overview of OpenAI Crawlers} Google-Extended crawls nothing and has no IP to verify: it is a permission over what Googlebot already crawls, and Google separates it from Search.⁠^{[2]Google Crawling Infrastructure. Google's common crawlers} Blocking GPTBot closes a door to a visitor; blocking Google-Extended changes the use made of what comes in through another door.

To go further, the Googlebot entry explains the crawler that does visit your store, the guide to what an AI crawler sees on your store goes agent by agent through what each one reads, and the reading path for this topic orders the rest of the terms.