# robots.txt for AI bots: check and write your rules

> Check which AI bots your robots.txt lets in, GPTBot, ClaudeBot and PerplexityBot among them, and write the lines that allow or block each one.

URL: https://clione.ai/tools/robots-txt-ai-bots/
Lang: en

If you run a store, or your clients' stores from an agency, this tool answers two questions: **which AI bots your [robots.txt](/glossary/robots-txt/) lets in today**, and which lines you need to write to let each one in, or keep it out. The first part reads your file the way a [crawler](/glossary/web-crawler/) that follows the standard reads it; the second writes the lines from what you choose. It all happens in your browser: the tool makes no requests, and nothing you paste leaves this page.

## How to use it

1. **Paste the live file.** Open `yourstore.com/robots.txt` in your browser, copy the whole file and paste it into the first field. Paste what your domain serves, not the platform's settings screen or Shopify's `robots.txt.liquid` template: the template's [Liquid](/glossary/shopify-liquid/) lines are not rules, and the tool does not interpret them.
2. **Choose the path.** By default it checks `/`, the home page. To see what happens on a [product page](/glossary/product-page/), type its path, like `/products/linen-shirt`.
3. **Press Check.** If you'd like to see a result first, "Load an example" pastes the file of a fictitious store that keeps [GPTBot](/glossary/gptbot/) and [ClaudeBot](/glossary/claudebot/) out and keeps everyone away from the cart and the checkout.
4. **Write your lines.** Under "Write the lines", choose Allow or Block for each agent and copy the result with the Copy button.

## How to read the result

The check returns one row per agent:

- **[User agent](/glossary/user-agent/)**: the name the agent looks for in your file, the one that goes after `User-agent:`.
- **Operator**: the company that runs it.
- **Group that applies**: "Its own group" if a `User-agent` line names it; "The \* group" if none names it and it falls into the general one; "No group: everything is allowed" if the file has neither.
- **Verdict**: Allowed or Blocked, for the path you typed.
- **The note in the last column** shows up on the agents a person launches from ChatGPT and from [Perplexity](/glossary/perplexity/), [ChatGPT-User](/glossary/chatgpt-user/) and Perplexity-User, because their companies say robots.txt may not apply to them or that they generally ignore it. There, a "Blocked" is what your file asks for, not what will happen.

Below the tool, under "The AI user agents this tool knows", is the list of the nine agents it covers, with their company, what each one is for and the page where its company documents it, with the date we read it. **An agent is only there if its company documents it on a page of its own**: none of them comes from a third-party list.

## How each agent picks its group

A robots.txt is read in groups: one or more `User-agent` lines and the rules below them. The standard, RFC 9309, sets how a crawler picks its own, and the tool applies those same rules:

- It looks for the group that names it, ignoring case. If there are several, it merges their rules into one. If none names it, it follows the `User-agent: *` group, and if that doesn't exist either, it has no rules.[^rfc9309]
- Of the rules whose path matches, the longest one wins, and if an `Allow` and a `Disallow` tie, the `Allow` applies.[^rfc9309] Google calls this using the least restrictive rule.[^google-robots-spec]
- In a path, `*` stands for any sequence of characters and a trailing `$` marks where the path ends.[^rfc9309]

The first rule has a consequence that is easy to miss: **an agent with its own group stops reading the general one**. In this fictitious example, [OAI-SearchBot](/glossary/oai-searchbot/) finds its own group and, with it, permission to enter `/cart` and `/checkout`:


```text
User-agent: *
Disallow: /cart
Disallow: /checkout

User-agent: OAI-SearchBot
Allow: /
```


The generator writes groups like the second one, one per agent, with `Allow: /` or `Disallow: /`, on purpose: that way the decision about each agent does not depend on the rest of the file. If your general group has `Disallow` lines you want to keep for everyone, copy them into each agent group you allow, and paste the final file into the check before you publish it.

## Three agents per company, three decisions

AI companies don't run one bot each but one per use, and each one is controlled separately:

| Operator   | Trains models                                                       | Searches for its answers                             | Visits a page because a person asked |
| ---------- | ------------------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------ |
| OpenAI     | GPTBot                                                              | OAI-SearchBot                                        | ChatGPT-User                         |
| Anthropic  | ClaudeBot                                                           | Claude-SearchBot                                     | Claude-User                          |
| Perplexity | None of its crawlers, according to its documentation                | [PerplexityBot](/glossary/perplexitybot/)            | Perplexity-User                      |
| Google     | [Google-Extended](/glossary/google-extended/), which does not crawl | [Googlebot](/glossary/googlebot/), outside this tool | Outside this tool                    |

At OpenAI, each setting is independent of the others: you can allow OAI-SearchBot to show up in ChatGPT's search answers and block GPTBot, the one that collects content to train its models.[^openai-bots] At Anthropic, ClaudeBot collects content that may be used for training, and Claude-SearchBot crawls to improve the quality of its search results.[^anthropic-bots] Perplexity says PerplexityBot exists to surface and link to sites in its results and is not used to train models.[^perplexity-bots]

Hence the most useful rule on this page: **blocking the training crawler does not take you out of [AI search](/glossary/ai-search/); blocking the search crawler does**. OpenAI says a site that opts out of OAI-SearchBot is not shown in ChatGPT's search answers,[^openai-bots] and Anthropic warns that blocking Claude-SearchBot may reduce your visibility in its results.[^anthropic-bots] When it makes sense to keep each one out is a separate decision, and it is covered in [AI bot blocking](/glossary/ai-bot-blocking/).

**Google-Extended is not a crawler.** It has no user agent of its own: Google crawls with its usual agents and reads that name in your robots.txt only as a control, which decides whether what it crawled can be used to train [Gemini](/glossary/gemini/) models and to ground their answers. According to Google, it does not affect your inclusion in Search and is not a ranking signal.[^google-common-crawlers] What you ask of Googlebot, on the other hand, affects Search and all of its features,[^google-common-crawlers] and to appear as a link in its AI features, Google asks that the page be indexed and eligible to be shown with a snippet.[^google-ai-features] Googlebot is not in the tool because it is not an AI bot, and blocking it is a different conversation.

**The agents a person launches may not follow the file.** OpenAI says robots.txt rules may not apply to ChatGPT-User, because its visits are requested by someone in ChatGPT,[^openai-bots] and Perplexity says Perplexity-User generally ignores them, for the same reason.[^perplexity-bots] Anthropic says its bots respect robots.txt and support `Crawl-delay`, a directive that is not part of the standard.[^anthropic-bots]

## robots.txt is a request, not a wall

The standard says so itself: these rules are not a form of access authorization.[^rfc9309] Shopify puts it its own way: the rules in your robots.txt are guidance, and not every crawler follows them.[^shopify-robots] A `Disallow` is a request a well-behaved crawler honors, not a locked door.

It works the other way around too: **an `Allow` does not open the door of your CDN or your firewall**. Google asks that crawling be allowed in robots.txt and also in any CDN or hosting.[^google-ai-features] If you want to really block an agent, the place to do it is the firewall, and there a name is not enough: Perplexity recommends checking the user agent and also that the IP is on the lists it publishes.[^perplexity-bots] Blocking by IP has its own limit: Anthropic warns that it may not work well or guarantee a permanent opt-out, because it stops them from reading your robots.txt.[^anthropic-bots]

A change is not instant either. The standard allows the file to be cached for up to 24 hours,[^rfc9309] and OpenAI estimates about 24 hours for its search to reflect a change.[^openai-bots]

## Where the lines go in Shopify and BigCommerce

**Shopify.** You change the file with the `robots.txt.liquid` template: in the theme's code editor, you add a new template of the robots type and save it in the published theme. Shopify warns that this is an unsupported customization and that getting it wrong can cost you all of your traffic.[^shopify-robots] Its developer documentation asks you not to replace the template with plain text: keep the Liquid objects that output the default rules and add your groups below them, which is how its own example allows or blocks a crawler.[^shopify-robots-liquid] That is where the generator's lines go, with the general group's `Disallow` lines copied in if you want to keep them. And one thing this file does not touch: blocking AI crawlers in robots.txt or at the network level only affects what is discovered on the open web, not the product data that [Shopify Catalog](/glossary/shopify-catalog/) sends to any [agentic storefronts](/glossary/agentic-storefronts/) you have turned on.[^shopify-robots]

**BigCommerce.** You edit the file in `Settings › Website`, in the Search Engine Robots section, and with Multi-Storefront, per storefront. What you see there is not the whole file: BigCommerce applies separate rules to AI and high-volume crawlers, with `Crawl-delay: 10` and blocks on sensitive pages, which don't appear in the settings but do appear in your `/robots.txt`. By default, you don't need to add anything for AI bots to crawl the store.[^bigcommerce-robots] That is why, on BigCommerce more than anywhere, **paste the live file into the check, not the one in the settings**.

## What the tool can't tell you

- **It does not download your file.** Browsers restrict the requests a script makes to another domain,[^mdn-cors] so it works with what you paste.
- **It tells you what your file asks for, not what each bot did.** Whether an agent respected your rules only shows in your server logs, with its IP checked against its company's list. The guide [what an AI crawler sees in your store](/guides/what-an-ai-crawler-sees/) explains how.
- **It does not see your CDN or your firewall**, which may block an agent your file lets through.
- **It checks one path at a time.** A store with different rules for product pages, collections and internal search needs one check per path.
- **It does not read `Crawl-delay` or `Sitemap`**, and it does not know how your server responds when the file is requested. That matters: with a 4xx error a crawler may access everything, and with a 5xx the standard asks it to assume it can access nothing.[^rfc9309]
- **It only knows nine agents.** A bot that is not on the list does not show up in the result, even if your file names it.