# What is GPTBot and what happens if you block it?

> What GPTBot is, the crawler OpenAI uses to collect content for training its models, what changes if you block it, and how to verify its visits.

URL: https://clione.ai/glossary/gptbot/
Lang: en

GPTBot is not a model or a version of ChatGPT: it is a [web crawler](/glossary/web-crawler/), the program OpenAI uses to travel the web and gather content to train its models. For a store, the question this name brings is almost always the same: **do I block it or not?** And the answer starts by separating two things that often get mixed up: your catalog being used to train a model, and your store showing up when someone searches in [ChatGPT Search](/glossary/chatgpt-search/). GPTBot only has to do with the first.

## How it works

OpenAI uses several crawlers and agents, and lets each site manage them separately in [robots.txt](/glossary/robots-txt/). According to its documentation, **each setting is independent of the others**: you can allow [OAI-SearchBot](/glossary/oai-searchbot/) so you show up in search results and block GPTBot to signal that what it crawls should not be used to train its foundation models. If you allow both, OpenAI may use a single crawl for both purposes and avoid repeating visits.[^openai-bots]

About GPTBot specifically, OpenAI says two things: that it crawls content that may be used to train its generative AI foundation models, to make them more useful and safer, and that **blocking it signals that a site's content should not be used for that training**.[^openai-bots] Note the verb: blocking "signals." robots.txt is a signal that an honest crawler respects, and the standard warns that its rules are not a form of access authorization.[^rfc9309]

GPTBot's request carries this [user agent](/glossary/user-agent/), in the version OpenAI documents as of October 2026:[^openai-bots]

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
```

In robots.txt the short name, `GPTBot`, is enough. The standard says each crawler looks for the group that names it, without distinguishing upper and lower case, and only if it finds none does it follow the `User-agent: *` group.[^rfc9309] So a store that does not mention GPTBot in its file applies the same rules to it as to any other bot.

OpenAI publishes the IP ranges GPTBot crawls from in a JSON file, `openai.com/gptbot.json`.[^openai-bots] The file carries a creation date and a list of network prefixes: the version we read on October 6, 2026 was dated September 22, 2026 and had 18 IPv4 prefixes.[^gptbot-ips] **The list changes**, so you look it up when you verify, you do not copy it once.

GPTBot has equivalents at other companies, and each one is blocked by its own name. Anthropic uses [ClaudeBot](/glossary/claudebot/) to collect content that improves the usefulness and safety of its generative AI models.[^anthropic-bots] Google has no separate crawler for that: it has [Google-Extended](/glossary/google-extended/), a name used in robots.txt to decide whether what Google crawls can be used to train future versions of [Gemini](/glossary/gemini/), and which according to Google **does not affect your presence in Search** and is not a ranking signal.[^google-common-crawlers]

Platforms play a part too. On Shopify, blocking AI crawlers in robots.txt or at the network level only affects what is discovered on the open web, and **it does not stop the product data that [Shopify Catalog](/glossary/shopify-catalog/) sends to the [agentic storefronts](/glossary/agentic-storefronts/)** you have turned on.[^shopify-robots] On BigCommerce, the platform applies its own rules, with a 10-second delay, to AI crawlers and high-volume ones, which you only see if you open the published file.[^bigcommerce-robots]

## In a store

A mountain gear store decides it does not want to hand over its catalog to train models, but does want to show up in ChatGPT search. It is a **fictitious example**. Its robots.txt:

```text
User-agent: *
Disallow: /cart
Disallow: /checkout

User-agent: GPTBot
Disallow: /
```

GPTBot finds its group and stops reading the general one: it cannot request anything. OAI-SearchBot has no group of its own, so it follows the asterisk's and can request the [product pages](/glossary/product-page/), but not the cart or the checkout.[^rfc9309] What the store is doing is separating the two uses, which is exactly what OpenAI proposes.[^openai-bots]

A day later, this line shows up in the server log (made up, with an IP reserved for documentation):

```text
192.0.2.33 [07/Oct/2026:03:12:44 +0200] "GET /products/backpack-40l HTTP/1.1" 200 "...compatible; GPTBot/1.4; +https://openai.com/gptbot"
```

Before concluding that GPTBot is skipping the rule, there are two questions. The first, whether the IP is on OpenAI's list: anyone can write GPTBot into a request, and **if the IP is not in `gptbot.json`, the visit does not come from any of the addresses OpenAI declares for GPTBot**.[^gptbot-ips] The second, when the change was published: the standard lets a crawler use its cached copy of the file for up to 24 hours.[^rfc9309]


**Blocking GPTBot does not take you out of ChatGPT search.** The agent that shows sites in ChatGPT's search features is OAI-SearchBot, and OpenAI says expressly that the two are managed separately.[^openai-bots] If you want to show up in those answers, the one you must not block is OAI-SearchBot. It does not work the other way around either: allowing GPTBot does not make you appear in them. And there is a third agent, [ChatGPT-User](/glossary/chatgpt-user/), that visits pages when a person asks ChatGPT to, and to which **robots.txt rules may not apply**.[^openai-bots]


## How to check it

1. **Open `yourdomain.com/robots.txt` and look for a group that names GPTBot.** If there is none, the asterisk group applies to it.[^rfc9309]
2. **On BigCommerce, read the published file**, not the settings screen: the rules the platform applies to AI bots only show up there.[^bigcommerce-robots]
3. **Search for GPTBot in your server or CDN logs** and note which paths it requests and how often.
4. **Check each IP against `openai.com/gptbot.json`** on the same day. An IP outside those ranges is not one OpenAI declares for its crawler.[^openai-bots][^gptbot-ips]
5. **After a change, wait at least a day** before drawing conclusions, because the file can stay cached for up to 24 hours.[^rfc9309]

## Related concept: OAI-SearchBot

Both belong to OpenAI and both are managed in robots.txt, but they do different jobs. GPTBot crawls content that may be used to train models; OAI-SearchBot is for showing sites in ChatGPT's search results, and **sites that block it do not appear in the answers of that search**, though they can still show up as navigation links.[^openai-bots] The combination OpenAI gives as an example, allowing one and blocking the other, separates visibility in its search from the use of your content for training.[^openai-bots]

To go on, the guide to [what an AI crawler sees on your store](/guides/what-an-ai-crawler-sees/) goes agent by agent through what each one reads, and the [reading path](/learn/crawling-and-indexing/) for this topic puts the rest of the terms in order.