What is AI bot blocking, and when does it make sense?

"Block AI" sounds like one decision, and for a store it is at least three: whether your catalog can be used to train models, whether you want to show up in the answers of assistants that search the web, and whether you let through an agent that visits your product page because a person asked it to. Each one is made with a different agent, and blocking the wrong one can take you out of the answers you wanted. The syntax of the rules is in the robots.txt entry; this one covers the decision, and the layers where it is applied.

What can be blocked

AI companies split their agents by function, and each one identifies itself with its own user agent name. OpenAI uses this as its own example: you can allow OAI-SearchBot so you appear in ChatGPT search results and block GPTBot to signal that what it crawls should not be used for training.⁠^{[1]OpenAI. Overview of OpenAI Crawlers}

The table's sources are each company's documentation.⁠^{[1]OpenAI. Overview of OpenAI Crawlers}⁠^{[2]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?}⁠^{[3]Perplexity. Perplexity Crawlers} Google is the odd one out. Google-Extended is not a crawler: it is a name used in robots.txt to decide whether what Google already crawls can be used to train future generations of Gemini. It does not affect your inclusion in Google Search and it is not a ranking signal, and Google crawls with its usual agents.⁠^{[4]Google Crawling Infrastructure. Google's common crawlers}

The bottom row is the one that surprises people. OpenAI says that because ChatGPT-User visits are initiated by a person, robots.txt rules may not apply, and that OAI-SearchBot is what you use to opt in or out of search.⁠^{[1]OpenAI. Overview of OpenAI Crawlers} Perplexity says Perplexity-User, for the same reason, generally ignores robots.txt.⁠^{[3]Perplexity. Perplexity Crawlers}

The three layers where blocking happens

robots.txt: a request. This is the layer well-behaved crawlers read, and the standard itself is clear about it: its rules are not a form of access authorization and do not replace a security measure.⁠^{[5]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} Shopify puts it in its own words: the rules are guidance, and not every crawler is obliged to follow them.⁠^{[6]Shopify Help Center. Editing robots.txt.liquid} Changes are not instant either: OpenAI and Perplexity estimate up to about 24 hours for their systems to apply them.⁠^{[1]OpenAI. Overview of OpenAI Crawlers}⁠^{[3]Perplexity. Perplexity Crawlers}

The network: a real block. A CDN or a web application firewall (WAF) can reject the request before it reaches your store. Cloudflare, for example, offers AI Crawl Control on all its plans: it shows which AI services visit your site, lets you allow or block each crawler separately and flags which ones ignore your robots.txt.⁠^{[7]Cloudflare Docs. AI Crawl Control} When you block one, it creates or updates a firewall rule that answers with a 403 by default.⁠^{[8]Cloudflare Docs. Manage AI crawlers} On July 1, 2025, Cloudflare announced it would block AI crawlers by default unless they paid creators for the content.⁠^{[9]Cloudflare Blog (Matthew Prince). Content Independence Day: no AI crawl without compensation!} And its older setting, Block AI bots, which blocks verified bots classified as training and other unverified ones that behave the same way, is retired on September 15, 2026, replaced by finer categories: search, agent and training.⁠^{[10]Cloudflare Docs. Block AI bots}

Blocking on the network means knowing who is who, because a bot's name can be forged. Google warns about this and explains how to verify its crawlers by their IP ranges and reverse DNS.⁠^{[4]Google Crawling Infrastructure. Google's common crawlers} OpenAI and Perplexity publish each agent's IPs in JSON files, and Perplexity recommends combining the name and the IP when you configure a WAF.⁠^{[1]OpenAI. Overview of OpenAI Crawlers}⁠^{[3]Perplexity. Perplexity Crawlers} Anthropic warns about the opposite limit: blocking its IPs may not work well or guarantee exclusion, because it stops it from reading your robots.txt.⁠^{[2]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?}

The platform: what comes already done. Shopify manages bots on its network without you doing anything and advises against putting a proxy in front of the store. It also sets an important limit: blocking AI crawlers in robots.txt or on the network only affects what is discovered on the open web, not the product data that Shopify Catalog sends to the agentic storefronts you have turned on; that is controlled in their own settings.⁠^{[6]Shopify Help Center. Editing robots.txt.liquid} BigCommerce applies separate rules to AI and high-volume crawlers, with a 10-second delay between requests and the cart, checkout, account and filtered listings closed off; you do not see them in the settings, only in the published robots.txt.⁠^{[11]BigCommerce Support. Understanding the Robots.txt File}

In a store

A fictional example: a mountain gear store that wants to appear in assistants' answers and does not want its catalog used to train models.

The table says something a blog's list does not: closing training and staying in search is not a contradiction, because they are different agents.⁠^{[1]OpenAI. Overview of OpenAI Crawlers} The typical mistake is copying an "all AI bots" list that lumps the search agents in with the rest.

How to check it

1. Open your published robots.txt and note which agents in the table have a group of their own and what that group says about your product pages. On BigCommerce, look at the live file, not the settings.⁠^{[11]BigCommerce Support. Understanding the Robots.txt File}
2. Check the CDN or the firewall. If you use Cloudflare, open AI Crawl Control and look at the action set for each crawler; if you had the old Block AI bots setting on, check which categories it moved to.⁠^{[8]Cloudflare Docs. Manage AI crawlers}⁠^{[10]Cloudflare Docs. Block AI bots}
3. Look in the logs for 403 responses to AI agents and check each one's IP against the lists its company publishes before trusting the name.⁠^{[1]OpenAI. Overview of OpenAI Crawlers}⁠^{[3]Perplexity. Perplexity Crawlers}
4. On Shopify, review the agentic storefronts settings separately: neither robots.txt nor the network decides what data goes out that way.⁠^{[6]Shopify Help Center. Editing robots.txt.liquid}
5. After a change, wait a day before drawing conclusions: that is the window OpenAI and Perplexity give.⁠^{[1]OpenAI. Overview of OpenAI Crawlers}⁠^{[3]Perplexity. Perplexity Crawlers}

Related concept: pay per crawl

Between letting through and blocking there is a third option: charging. Cloudflare's pay per crawl lets you set a price per zone, and a crawler that does not show intent to pay gets a 402 Payment Required with the price instead of the content. It is in closed beta.⁠^{[12]Cloudflare Docs. What is Pay Per Crawl} Blocking answers who can come in; payment answers how much it costs, and both rest on the same thing: knowing which bot is really calling, and not just what it says its name is.

To go further, the guide to what an AI crawler sees in your store checks it agent by agent, and the reading path for this topic orders the rest of the terms.