# What is robots.txt and what it does for an online store

> What the robots.txt file is, how Googlebot, GPTBot and OAI-SearchBot read it, why it can't remove a page from the index and how to review your store's.

URL: https://clione.ai/glossary/robots-txt/
Lang: en

robots.txt has nothing to do with robotics, or with a file you have to install: it is plain text that lives at the root of your domain and that any polite [web crawler](/glossary/web-crawler/) reads before requesting a page. For a store it matters for two reasons. It is **where you decide which bots may request your product pages**, Google's and each AI assistant's, one by one. And it is a small file with a lot of power: Shopify warns that misusing its customization **can cost you all your traffic**.[^shopify-robots]

## How it works

The standard is the Robots Exclusion Protocol, which Martijn Koster defined in 1994 and which the IETF published as RFC 9309 in 2022. The rules have to be in a file called `/robots.txt`, all lowercase, at the top level of the site.[^rfc9309] Google adds a nuance that is often forgotten: **the rules only apply to the host, protocol and port where the file lives**, so every subdomain needs its own.[^google-robots-spec]

The file is organized in groups. Each group starts with one or more `User-agent` lines, which name a bot, and continues with its `Allow` and `Disallow` rules. Each crawler looks for the group that names it, case-insensitively, and if none does it obeys the `User-agent: *` group.[^rfc9309] Google puts it more clearly: only one group applies to a crawler, the one with the most specific name that matches it, and **the specific group and the general one are not combined**.[^google-robots-spec] In other words: if you give a bot a group of its own, it stops reading the asterisk's rules.

Within a group, the most specific rule wins, the one that matches the most characters of the path, and if an `Allow` and a `Disallow` tie, the `Allow` applies.[^rfc9309] In Google's example, with `allow: /p` and `disallow: /`, the URL `/page` can be crawled, because the first rule is more specific. Paths are case-sensitive, and they support two wildcards: `*` for any sequence of characters and `$` for the end of the URL.[^google-robots-spec]

Google only recognizes four fields: `user-agent`, `allow`, `disallow` and `sitemap`. **`Crawl-delay` isn't among them and Google ignores it.**[^google-robots-spec] Anthropic, on the other hand, says it supports that non-standard extension.[^anthropic-bots] The `Sitemap` line points to your [XML sitemap](/glossary/xml-sitemap/) with a full URL and doesn't depend on any group: every crawler can follow it.[^google-robots-spec]

Three operational details weigh more than the syntax:

- **The response code counts.** If the file returns a 4xx, the crawler may request anything. If it returns a 5xx, the standard tells it to assume it may request nothing.[^rfc9309] Google, in that case, stops crawling the site for the first 12 hours while it keeps trying to read the file, and then uses the last saved copy for up to 30 days.[^google-robots-spec]
- **Changes aren't immediate.** The standard says a crawler shouldn't use the cached copy for more than 24 hours,[^rfc9309] and OpenAI estimates about 24 hours for its search to adjust to a change.[^openai-bots]
- **Size has a limit.** Google reads at most 500 KiB and ignores anything past that.[^google-robots-spec]

Finally, what the file isn't. The standard itself says **these rules are not a form of access authorization** and don't replace a security measure,[^rfc9309] and Shopify describes them as guidance: not every crawler is obliged to follow them.[^shopify-robots]

## In a store

AI companies use several agents, and each one is allowed or blocked separately. OpenAI gives an example: you can let [OAI-SearchBot](/glossary/oai-searchbot/) through so you appear in [ChatGPT search](/glossary/chatgpt-search/) and block [GPTBot](/glossary/gptbot/) to indicate that what it crawls shouldn't be used to train its models.[^openai-bots] Google has a similar control, [Google-Extended](/glossary/google-extended/), which decides whether what it crawls can be used to train future versions of [Gemini](/glossary/gemini/), and which, according to Google, doesn't affect your presence in Search or your ranking.[^google-common-crawlers] The full table of agents is in the [GEO entry](/glossary/geo/).

Here is what the file might look like for a store that wants to keep the cart and sorted collections out, and not give up its content to train models. It is a **fictional example**:

```text
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /collections/*sort_by*

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://example-store.com/sitemap.xml
```

What each part does. The first group applies to any bot that doesn't have its own: [Googlebot](/glossary/googlebot/), OAI-SearchBot and the rest can request the product pages, but not the cart or the reordered collections, which only multiply [faceted navigation](/glossary/faceted-navigation/) URLs. The second group names three training agents and closes the whole site to them. Notice what doesn't appear: **OAI-SearchBot has no group of its own, on purpose**. If you gave it one with an `Allow: /`, it would stop reading the asterisk and could request `/cart` and `/checkout` too.[^rfc9309][^google-robots-spec]

On Shopify you don't start from zero. Every store has a default robots.txt that blocks, among other paths, the admin panel, the cart, checkout, the customer account, orders and sorted or filtered collections. To change it you create the `robots.txt.liquid` template in the theme, and Shopify warns that it is an unsupported customization. Blocking AI crawlers there only affects what is discovered on the open web: **it doesn't stop the product data that [Shopify Catalog](/glossary/shopify-catalog/) sends to the [agentic storefronts](/glossary/agentic-storefronts/)** you have turned on.[^shopify-robots]

On BigCommerce the file is edited at `Settings › Website › Search Engine Robots`. What you see there isn't everything: the platform applies separate rules to AI crawlers and high-volume ones, with a 10-second delay and sensitive pages blocked, which only show up if you open the published file. By default you don't need to add anything for AI bots to crawl the store.[^bigcommerce-robots]


**Allowing crawling isn't asking for [indexing](/glossary/indexing/), and blocking it isn't leaving the index.** An `Allow` only says the crawler may request the URL; what gets stored is decided by the search engine afterwards. And a page blocked in robots.txt can still show up in Google if other sites link to it, without a description.[^google-robots-intro] To keep it out there is [`noindex`](/glossary/noindex/), which only works if the page isn't blocked: if the crawler can't read it, **it will never see the instruction**.[^google-noindex]


## How to check it

1. **Open `yourdomain.com/robots.txt`** and check that it answers with a 200. Do the same on every subdomain that serves pages, because each one carries its own rules.[^google-robots-spec]
2. **For each bot you care about, look for the group that names it.** If there isn't one, the asterisk's applies. Then see what that group says about the path of your product pages, for example `/products/`.
3. **Open the robots.txt report in [Search Console](/glossary/google-search-console/).** It shows the files Google found on the top 20 hosts of your site, when it last read them and any errors or warnings. If you have fixed something urgent, you can ask it from there to read the file again.[^gsc-robots-report]
4. **Test a specific URL with URL Inspection**, which tells you whether Google is blocked from accessing it.[^gsc-robots-report]
5. **On Shopify or BigCommerce, read the published file, not the settings screen**: on BigCommerce, the rules for AI bots only show up there.[^bigcommerce-robots]

## Related concept: noindex

robots.txt controls crawling; `noindex` controls indexing. The `noindex` rule goes in a `<meta name="robots">` tag or in the `X-Robots-Tag` HTTP header, and when Googlebot reads it, Google drops the page from its results even if other sites link to it.[^google-noindex] That is why they aren't combined lightly: **if you block the URL in robots.txt, the crawler never gets to read the `noindex`** and the page can keep appearing.[^google-noindex] The right order for retiring a [product page](/glossary/product-page/) is `noindex` first with crawling open, and the block afterwards, if needed. When it makes sense to close the door to each AI bot is a different decision, and you'll find it in [AI bot blocking](/glossary/ai-bot-blocking/).

To keep going, the guide to [what an AI crawler sees on your store](/guides/what-an-ai-crawler-sees/) reads the file agent by agent, and the [crawling and indexing reading path](/learn/crawling-and-indexing/) puts the other terms in order.