# Stop your store from blocking AI bots, Cloudflare included

> If your store returns a 403, a 429 or a challenge to AI crawlers, see which layer decides it and which setting to change, from Cloudflare to robots.txt.

URL: https://clione.ai/guides/store-blocking-ai-crawlers/
Lang: en

Before you touch anything, decide who you want to let in, because **[blocking AI bots](/glossary/ai-bot-blocking/) isn't a single decision**, not in your [robots.txt](/glossary/robots-txt/) and not in the firewall. Each [crawler](/glossary/web-crawler/) comes in for a different job: OpenAI, for example, separates the agent that searches from the one that trains and from the one that visits a page because a person asked it to.[^openai-bots] Anthropic and [Perplexity](/glossary/perplexity/) document their agents the same way, each with its own job:[^anthropic-bots][^perplexity-bots]

| What it comes in for     | Agents                                                                                                 | For a store that wants to show up in answers                                                                                                                                                       |
| ------------------------ | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Search and cite          | [OAI-SearchBot](/glossary/oai-searchbot/), Claude-SearchBot, [PerplexityBot](/glossary/perplexitybot/) | Let them in                                                                                                                                                                                        |
| Visits a person asks for | [ChatGPT-User](/glossary/chatgpt-user/), Claude-User, Perplexity-User                                  | Let them in                                                                                                                                                                                        |
| Search and train at once | [Googlebot](/glossary/googlebot/), [Bingbot](/glossary/bingbot/), Applebot                             | Always let them in: these are your presence in Google Search, in Bing and in search across Siri, Spotlight and Safari |
| Train only               | [GPTBot](/glossary/gptbot/), [ClaudeBot](/glossary/claudebot/)                                         | Your call: blocking them doesn't remove you from any search                                                                                                                                        |

The last two rows follow Cloudflare's classification,[^cf-mixed-use] and Apple uses what Applebot crawls for the search built into Siri, Spotlight and Safari.[^apple-bot]

## Step 1: read which layer said no

Before you change a setting, look at what the agent received. Request a [product page](/glossary/product-page/) with each agent's [user agent](/glossary/user-agent/) and look for its real visits in your logs, as [steps 4 and 5 of the guide to what an AI crawler sees in your store](/guides/what-an-ai-crawler-sees/) explain. What you find tells you where to look:

| What the agent receives                      | Who can say it                                                                                                                                            | Where it's fixed |
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| A 403                                        | A CDN or firewall rule. Cloudflare's AI Crawl Control responds with a 403 when it blocks a crawler; on paid plans you can change it to a 402.[^cf-manage] | Steps 2, 3 and 5 |
| A 429                                        | A rate limit: it's the code Cloudflare returns by default.[^cf-rl-params]                                                                                 | Step 3           |
| A challenge page instead of the product page | A bot manager that challenges the visitor, such as Bot Fight Mode.[^perplexity-bots]                                                                      | Steps 3 and 5    |
| No visits, and a `Disallow` in the file      | Your robots.txt, or the rules Cloudflare puts in front of it.[^cf-managed-robots]                                                                         | Step 4           |
| The store's password page                    | Shopify's private mode, which closes the store to visitors and search engines.[^shopify-prefs]                                                            | Step 6           |

The 403 costs the most. When Googlebot gets a 403 on a URL Google already had in its index, it drops that URL from the index. A 429 or a 503, by contrast, makes it crawl more slowly, though if they last, indexed URLs drop out too.[^google-http] And **a Disallow produces no code at all**: an agent that respects robots.txt never requests the page.

## Step 2: Cloudflare, the AI policy and AI Crawl Control

**In Cloudflare, setting training to Block also blocks Googlebot, Bingbot and Applebot.** Cloudflare counts them as mixed-use crawlers, which serve search and training at once, and since September 15, 2026 its Block and Block on pages with ads options apply to them too. Blocking them takes you out of search, not only out of training.[^cf-mixed-use] It's the exact opposite of what happens with Google's own controls: there, training is decided with [Google-Extended](/glossary/google-extended/), which doesn't affect your inclusion in Search.[^google-common-crawlers]

For training, the option is **Disallow**. It publishes your preference in robots.txt and leaves those mixed-use crawlers allowed for search.[^cf-mixed-use] Whether Disallow also stops training crawlers at Cloudflare's edge, except the mixed-use ones, depends on Bot Preference Sync: turn it on in item 3 of the list below.[^cf-policies] **This is the call our reviewers sign off on: for training, Disallow, not Block.** The difference between the two options is the mixed-use crawlers, and a store that wants to sell needs those to get in. Cloudflare points out one limit: until Microsoft supports that preference in robots.txt, which it expects in early 2027, Disallow doesn't tell Bing that you don't want it training on your content.[^cf-mixed-use]

The defaults changed several times in 2026. On July 1, every plan, the free one included, got separate controls for Search, Agent and Training.[^cf-changelog] On September 15, Cloudflare retired the single Block AI bots toggle in favor of those controls, and new sign-ups got presets that depend on whether the site makes its money from ads.[^cf-mixed-use] And the current documentation describes yet another starting point.[^cf-policies] Don't assume which one your zone has: open it and look.

1. In the dashboard, go to Security › Settings, filter by Bot traffic and open AI Crawler and Agent access policies.[^cf-policies]
2. Set Search and Agent to Allow (do not block). For Personal Agents, Cloudflare recommends Allow.[^cf-policies]
3. Set Training to Disallow, the option that only exists for that category, and turn on Enable Bot Preference Sync on the same screen.[^cf-policies][^cf-managed-robots]
4. Then, in AI Crawl Control, review each crawler's action. You can allow or block them one by one, and each block creates or updates a custom WAF rule in the zone.[^cf-manage] If a search agent still gets a 403, that rule is the first suspect.

## Step 3: Bot Fight Mode and rate limits

**Bot Fight Mode can't be skipped with a custom WAF rule or with Page Rules.** It doesn't run on the rules engine, so the Skip, Bypass and Allow actions don't affect it.[^cf-bfm][^cf-skip] Perplexity warns about this in its documentation: Bot Fight Mode can keep challenging its agents even if you write rules to let them through.[^perplexity-bots] That leaves you three ways out: turn it off, for the whole domain; create IP Access rules, because Bot Fight Mode doesn't kick in if one of them matches the request first; or move to Super Bot Fight Mode, which runs on the rules engine and accepts Skip rules, as the next paragraph shows.[^cf-bfm] For IP Access rules you need the IPs each company publishes, such as `searchbot.json` or `chatgpt-user.json` at OpenAI,[^openai-bots] and Perplexity recommends updating those rules automatically, because the ranges change.[^perplexity-bots]

Super Bot Fight Mode splits traffic into three groups, Definitely automated, Likely automated and Verified bots, and each one gets its own response. Set Verified bots to Allow and, for any other exception, create a custom rule with the Skip action, which lets the request skip that phase and carry on through the rest of your security.[^cf-sbfm][^cf-skip] To Cloudflare, a verified bot is one that identifies itself honestly, with a [Web Bot Auth](/glossary/web-bot-auth/) signature, with a published IP list and a stable user agent, or with reverse DNS, and that doesn't abuse the access this gives it.[^cf-verified]

Rate limits respond with a 429 by default.[^cf-rl-params] Cloudflare warns that applying them to verified bots can affect your rankings, and on any plan the rule's expression can use the Verified Bot field to leave them out.[^cf-rl]

## Step 4: robots.txt, yours and the one Cloudflare adds

The robots.txt an agent reads may not be the one you uploaded. With Bot Preference Sync, Cloudflare generates rules from your search, agent and training policies and **puts them in front of your file**, which is still there, underneath.[^cf-managed-robots][^cf-bps] If all three policies are on Allow, the file goes out unchanged. As soon as any of the three, training included, is on Block or Block on pages with ads, or training is on Disallow, it adds its rules.[^cf-managed-robots] It comes on by default for new customers.[^cf-bps]

So read the live file, at `/robots.txt` on your domain, not the copy on your server. If a search agent shows up there with `Disallow: /` and you didn't write it, the cause is in the step 2 policies. The [robots.txt tool for AI bots](/tools/robots-txt-ai-bots/) writes and checks the lines that allow or block each agent, with the rule for which group each one reads.

## Step 5: other firewalls and bot managers

If Cloudflare isn't in front of your store, the setting has a different name but the idea is the same: find the AI category and see what it does with each type of agent.

- **AWS WAF.** In the Bot Control rule group, the CategoryAI rule blocks AI bots **whether they're verified or not**, and only requests that arrive with a validated Web Bot Auth signature get through.[^aws-bot-control] If a verified search agent is being turned away, start with that rule.
- **Akamai.** It separates three categories, training crawlers, search crawlers and agents that request content in real time, and lets you set a policy for each: limit training and allow search, for example.[^akamai-ai]
- **Fastly.** You choose the action for all the bots in a category, and each individual bot inherits it or overrides it with another.[^fastly-bots]
- **DataDome.** It classifies agents as AI Crawlers, AI Assistants, Autonomous Agents and [Agentic Browsers](/glossary/agentic-browser/). Most get an intent-based policy by default, which lets the traffic through unless it detects a fraud attempt; the exception is training crawlers, which generally come set to Block. You review this in Agentic Trust › Access Control before turning protection on.[^datadome]
- **Imperva.** To let a specific AI bot through, you create an exception with its Client App ID.[^imperva]

The same rule holds in all of them: **let agents through by verified category or by the IPs each company publishes, never by user agent alone**, because that string can be spoofed.[^aws-bot-control] Perplexity recommends combining the string with an IP check,[^perplexity-bots] and Google explains how to verify its crawlers with reverse DNS or with its lists of IP ranges.[^google-verify] A block by country counts too: Google crawls mostly from US IP addresses and, if it detects that you block requests from the US, it may try to crawl from other countries.[^google-crawlers]

## Step 6: what your platform decides

**Shopify.** While private mode is on, the store shows a password page and keeps out visitors and search engines.[^shopify-prefs] If the `Disallow` comes from your theme's `robots.txt.liquid` template, step 1 of the guide to what an AI crawler sees covers what the file contains by default and how to change it.[^shopify-robots]

**BigCommerce.** For stronger control over bots, BigCommerce points you to a service that manages access at the infrastructure level, such as Cloudflare.[^bigcommerce-robots] If you have one in place, steps 2 and 3 are yours.

## Step 7: confirm the fix

A robots.txt change doesn't take effect right away: OpenAI estimates about 24 hours for its systems to apply it, and Perplexity up to 24 hours.[^openai-bots][^perplexity-bots] After that day, go back to your logs as in step 5 of the guide to what an AI crawler sees: filter by the agent's name, check its IP against its company's list and make sure the response is a 200.

For Googlebot you have a direct window: the URL Inspection tool in [Search Console](/glossary/google-search-console/). "Test live URL" examines the page as it is at that moment, and its fields tell you whether Google could fetch it from your server and whether your robots.txt lets it crawl the page. If the block did drop URLs from the index, you can request [indexing](/glossary/indexing/) again from there, though requesting it doesn't guarantee they get back in.[^gsc-url-inspection] And if what Googlebot was getting was a 429 or a 503, Google gradually raises its crawl rate again once your server responds normally.[^google-http]


When we can't produce a store's report, it's sometimes because its site answers our crawler with a 403 at the home page itself. That 403 doesn't come from robots.txt: **a Disallow asks the agent not to come in; a 403 stops it**, and the call is made by a layer that comes earlier, the CDN, the firewall or the server. The standard says as much: robots.txt rules are not a form of access authorization.[^rfc9309] That's why Google asks for crawling to be allowed in robots.txt and also in any CDN or hosting infrastructure,[^google-ai-features] and why a store can have the file open to everyone and still shut the door on the agent it meant to let in. What gets fixed is the layer from step 1, not the file.


## What we checked

The menus, options and defaults in this guide come from each company's documentation, which we read on November 11, 2026. At Cloudflare, the pages on AI crawler and agent access policies, managed robots.txt and verified bots were updated on November 2, 2026; the one on rate limiting parameters, on November 4; the one on rate limiting rules, on August 25; the ones on Bot Fight Mode, Super Bot Fight Mode and the Skip action, on August 3; and the one on managing crawlers in AI Crawl Control, on July 28.[^cf-policies][^cf-managed-robots][^cf-verified][^cf-rl-params][^cf-rl][^cf-bfm][^cf-sbfm][^cf-skip][^cf-manage] What we say about Akamai and Imperva comes from their blogs, in posts from September 3 and June 30, 2026.[^akamai-ai][^imperva]

To write it, we didn't change any setting on a production zone, and we haven't yet repeated the steps on a test zone, turning on each option and noting which code each agent gets and which IP it comes from.
