# Blocking AI Bots: When It Makes Sense and How to Do It

> What AI bot blocking is, which agents a store should let through, and why robots.txt, the CDN and the platform each block different things.

URL: https://clione.ai/glossary/ai-bot-blocking/
Lang: en

"Block AI" sounds like one decision, and for a store it is at least three: whether your catalog can be used to train models, whether you want to show up in the answers of assistants that search the web, and whether you let through an agent that visits your [product page](/glossary/product-page/) because a person asked it to. Each one is made with a different agent, and **blocking the wrong one can take you out of the answers you wanted**. The syntax of the rules is in the [robots.txt](/glossary/robots-txt/) entry; this one covers the decision, and the layers where it is applied.

## What can be blocked

AI companies split their agents by function, and each one identifies itself with its own [_user agent_](/glossary/user-agent/) name. OpenAI uses this as its own example: you can allow [OAI-SearchBot](/glossary/oai-searchbot/) so you appear in [ChatGPT search](/glossary/chatgpt-search/) results and block [GPTBot](/glossary/gptbot/) to signal that what it crawls should not be used for training.[^openai-bots]

| Function                 | OpenAI                                  | Anthropic                         | [Perplexity](/glossary/perplexity/)           | Google                                                                                              |
| ------------------------ | --------------------------------------- | --------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| Train models             | GPTBot                                  | [ClaudeBot](/glossary/claudebot/) | (per the company, it does not crawl to train) | [Google-Extended](/glossary/google-extended/), with no [crawler](/glossary/web-crawler/) of its own |
| Search and cite          | OAI-SearchBot                           | Claude-SearchBot                  | [PerplexityBot](/glossary/perplexitybot/)     | Search's own agents                                                                                 |
| Visits a person requests | [ChatGPT-User](/glossary/chatgpt-user/) | Claude-User                       | Perplexity-User                               |                                                                                                     |

The table's sources are each company's documentation.[^openai-bots][^anthropic-bots][^perplexity-bots] Google is the odd one out. Google-Extended is not a crawler: it is a name used in robots.txt to decide whether what Google already crawls can be used to train future generations of [Gemini](/glossary/gemini/). **It does not affect your inclusion in Google Search and it is not a ranking signal**, and Google crawls with its usual agents.[^google-crawlers]

The bottom row is the one that surprises people. OpenAI says that because ChatGPT-User visits are initiated by a person, **robots.txt rules may not apply**, and that OAI-SearchBot is what you use to opt in or out of search.[^openai-bots] Perplexity says Perplexity-User, for the same reason, generally ignores robots.txt.[^perplexity-bots]

## The three layers where blocking happens

**robots.txt: a request.** This is the layer well-behaved crawlers read, and the standard itself is clear about it: its rules are **not a form of access authorization** and do not replace a security measure.[^rfc9309] Shopify puts it in its own words: the rules are guidance, and not every crawler is obliged to follow them.[^shopify-robots] Changes are not instant either: OpenAI and Perplexity estimate up to about 24 hours for their systems to apply them.[^openai-bots][^perplexity-bots]

**The network: a real block.** A CDN or a web application firewall (WAF) can reject the request before it reaches your store. Cloudflare, for example, offers AI Crawl Control on all its plans: it shows which AI services visit your site, lets you allow or block each crawler separately and flags which ones ignore your robots.txt.[^cf-aicc] When you block one, it creates or updates a firewall rule that answers with a 403 by default.[^cf-manage] On July 1, 2025, Cloudflare announced it would **block AI crawlers by default** unless they paid creators for the content.[^cf-independence] And its older setting, Block AI bots, which blocks verified bots classified as training and other unverified ones that behave the same way, is retired on September 15, 2026, replaced by finer categories: search, agent and training.[^cf-block-ai]

Blocking on the network means knowing who is who, because **a bot's name can be forged**. Google warns about this and explains how to verify its crawlers by their IP ranges and reverse DNS.[^google-crawlers] OpenAI and Perplexity publish each agent's IPs in JSON files, and Perplexity recommends combining the name and the IP when you configure a WAF.[^openai-bots][^perplexity-bots] Anthropic warns about the opposite limit: blocking its IPs may not work well or guarantee exclusion, because **it stops it from reading your robots.txt**.[^anthropic-bots]

**The platform: what comes already done.** Shopify manages bots on its network without you doing anything and advises against putting a _proxy_ in front of the store. It also sets an important limit: blocking AI crawlers in robots.txt or on the network **only affects what is discovered on the open web**, not the product data that [Shopify Catalog](/glossary/shopify-catalog/) sends to the [_agentic storefronts_](/glossary/agentic-storefronts/) you have turned on; that is controlled in their own settings.[^shopify-robots] BigCommerce applies separate rules to AI and high-volume crawlers, with a 10-second delay between requests and the cart, checkout, account and filtered listings closed off; you do not see them in the settings, only in the published robots.txt.[^bigcommerce-robots]

## In a store

A **fictional example**: a mountain gear store that wants to appear in assistants' answers and does not want its catalog used to train models.

| Agent                                          | What for             | Decision                       | Where                                                 |
| ---------------------------------------------- | -------------------- | ------------------------------ | ----------------------------------------------------- |
| GPTBot, ClaudeBot, Google-Extended             | Training             | Block                          | robots.txt; on the network, if it wants it enforced   |
| OAI-SearchBot, Claude-SearchBot, PerplexityBot | Search and citations | Let through                    | No group of its own in robots.txt and no network rule |
| ChatGPT-User, Claude-User, Perplexity-User     | Visits on request    | Let through                    | Check that the CDN is not stopping them               |
| Shopify Catalog AI channels                    | Product data         | Whatever it decides in Shopify | The _agentic storefronts_ settings                    |

The table says something a blog's list does not: **closing training and staying in search is not a contradiction**, because they are different agents.[^openai-bots] The typical mistake is copying an "all AI bots" list that lumps the search agents in with the rest.


**Blocking GPTBot does not take you out of ChatGPT search, and blocking OAI-SearchBot does.** OpenAI controls each agent separately: GPTBot is the training one and OAI-SearchBot is the search one.[^openai-bots] The other mix-up is believing that an `Allow` in robots.txt lets the bot through: if the CDN or the firewall blocks it, the request never arrives, whatever the file says.[^cf-manage]


## How to check it

1. **Open your published robots.txt** and note which agents in the table have a group of their own and what that group says about your product pages. On BigCommerce, look at the live file, not the settings.[^bigcommerce-robots]
2. **Check the CDN or the firewall.** If you use Cloudflare, open AI Crawl Control and look at the action set for each crawler; if you had the old Block AI bots setting on, check which categories it moved to.[^cf-manage][^cf-block-ai]
3. **Look in the logs for 403 responses to [AI agents](/glossary/ai-agent/)** and check each one's IP against the lists its company publishes before trusting the name.[^openai-bots][^perplexity-bots]
4. **On Shopify, review the _agentic storefronts_ settings separately**: neither robots.txt nor the network decides what data goes out that way.[^shopify-robots]
5. **After a change, wait a day** before drawing conclusions: that is the window OpenAI and Perplexity give.[^openai-bots][^perplexity-bots]

## Related concept: pay per crawl

Between letting through and blocking there is a third option: charging. Cloudflare's [pay per crawl](/glossary/pay-per-crawl/) lets you set a price per zone, and a crawler that does not show intent to pay gets a **402 Payment Required** with the price instead of the content. It is in closed beta.[^cf-ppc] Blocking answers who can come in; payment answers how much it costs, and both rest on the same thing: knowing which bot is really calling, and not just what it says its name is.

To go further, the guide to [what an AI crawler sees in your store](/guides/what-an-ai-crawler-sees/) checks it agent by agent, and the [reading path](/learn/crawling-and-indexing/) for this topic orders the rest of the terms.