What is GPTBot, OpenAI's crawler?

GPTBot is not a model or a version of ChatGPT: it is a web crawler, the program OpenAI uses to travel the web and gather content to train its models. For a store, the question this name brings is almost always the same: do I block it or not? And the answer starts by separating two things that often get mixed up: your catalog being used to train a model, and your store showing up when someone searches in ChatGPT Search. GPTBot only has to do with the first.

How it works

OpenAI uses several crawlers and agents, and lets each site manage them separately in robots.txt. According to its documentation, each setting is independent of the others: you can allow OAI-SearchBot so you show up in search results and block GPTBot to signal that what it crawls should not be used to train its foundation models. If you allow both, OpenAI may use a single crawl for both purposes and avoid repeating visits.⁠^{[1]OpenAI. Overview of OpenAI Crawlers}

About GPTBot specifically, OpenAI says two things: that it crawls content that may be used to train its generative AI foundation models, to make them more useful and safer, and that blocking it signals that a site's content should not be used for that training.⁠^{[1]OpenAI. Overview of OpenAI Crawlers} Note the verb: blocking "signals." robots.txt is a signal that an honest crawler respects, and the standard warns that its rules are not a form of access authorization.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}

GPTBot's request carries this user agent, in the version OpenAI documents as of October 2026:⁠^{[1]OpenAI. Overview of OpenAI Crawlers}

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

In robots.txt the short name, GPTBot, is enough. The standard says each crawler looks for the group that names it, without distinguishing upper and lower case, and only if it finds none does it follow the User-agent: * group.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} So a store that does not mention GPTBot in its file applies the same rules to it as to any other bot.

OpenAI publishes the IP ranges GPTBot crawls from in a JSON file, openai.com/gptbot.json.⁠^{[1]OpenAI. Overview of OpenAI Crawlers} The file carries a creation date and a list of network prefixes: the version we read on October 6, 2026 was dated September 22, 2026 and had 18 IPv4 prefixes.⁠^{[3]OpenAI. gptbot.json (GPTBot IP ranges)} The list changes, so you look it up when you verify, you do not copy it once.

GPTBot has equivalents at other companies, and each one is blocked by its own name. Anthropic uses ClaudeBot to collect content that improves the usefulness and safety of its generative AI models.⁠^{[4]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?} Google has no separate crawler for that: it has Google-Extended, a name used in robots.txt to decide whether what Google crawls can be used to train future versions of Gemini, and which according to Google does not affect your presence in Search and is not a ranking signal.⁠^{[5]Google Crawling Infrastructure. Google's common crawlers}

Platforms play a part too. On Shopify, blocking AI crawlers in robots.txt or at the network level only affects what is discovered on the open web, and it does not stop the product data that Shopify Catalog sends to the agentic storefronts you have turned on.⁠^{[6]Shopify Help Center. Editing robots.txt.liquid} On BigCommerce, the platform applies its own rules, with a 10-second delay, to AI crawlers and high-volume ones, which you only see if you open the published file.⁠^{[7]BigCommerce Support. Understanding the Robots.txt File}

In a store

A mountain gear store decides it does not want to hand over its catalog to train models, but does want to show up in ChatGPT search. It is a fictitious example. Its robots.txt:

User-agent: *
Disallow: /cart
Disallow: /checkout

User-agent: GPTBot
Disallow: /

GPTBot finds its group and stops reading the general one: it cannot request anything. OAI-SearchBot has no group of its own, so it follows the asterisk's and can request the product pages, but not the cart or the checkout.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} What the store is doing is separating the two uses, which is exactly what OpenAI proposes.⁠^{[1]OpenAI. Overview of OpenAI Crawlers}

A day later, this line shows up in the server log (made up, with an IP reserved for documentation):

192.0.2.33 [07/Oct/2026:03:12:44 +0200] "GET /products/backpack-40l HTTP/1.1" 200 "...compatible; GPTBot/1.4; +https://openai.com/gptbot"

Before concluding that GPTBot is skipping the rule, there are two questions. The first, whether the IP is on OpenAI's list: anyone can write GPTBot into a request, and if the IP is not in gptbot.json, the visit does not come from any of the addresses OpenAI declares for GPTBot.⁠^{[3]OpenAI. gptbot.json (GPTBot IP ranges)} The second, when the change was published: the standard lets a crawler use its cached copy of the file for up to 24 hours.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}

How to check it

1. Open yourdomain.com/robots.txt and look for a group that names GPTBot. If there is none, the asterisk group applies to it.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}
2. On BigCommerce, read the published file, not the settings screen: the rules the platform applies to AI bots only show up there.⁠^{[7]BigCommerce Support. Understanding the Robots.txt File}
3. Search for GPTBot in your server or CDN logs and note which paths it requests and how often.
4. Check each IP against openai.com/gptbot.json on the same day. An IP outside those ranges is not one OpenAI declares for its crawler.⁠^{[1]OpenAI. Overview of OpenAI Crawlers}⁠^{[3]OpenAI. gptbot.json (GPTBot IP ranges)}
5. After a change, wait at least a day before drawing conclusions, because the file can stay cached for up to 24 hours.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}

Related concept: OAI-SearchBot

Both belong to OpenAI and both are managed in robots.txt, but they do different jobs. GPTBot crawls content that may be used to train models; OAI-SearchBot is for showing sites in ChatGPT's search results, and sites that block it do not appear in the answers of that search, though they can still show up as navigation links.⁠^{[1]OpenAI. Overview of OpenAI Crawlers} The combination OpenAI gives as an example, allowing one and blocking the other, separates visibility in its search from the use of your content for training.⁠^{[1]OpenAI. Overview of OpenAI Crawlers}

To go on, the guide to what an AI crawler sees on your store goes agent by agent through what each one reads, and the reading path for this topic puts the rest of the terms in order.