What is the robots.txt file and what is it for?

robots.txt has nothing to do with robotics, or with a file you have to install: it is plain text that lives at the root of your domain and that any polite web crawler reads before requesting a page. For a store it matters for two reasons. It is where you decide which bots may request your product pages, Google's and each AI assistant's, one by one. And it is a small file with a lot of power: Shopify warns that misusing its customization can cost you all your traffic.⁠^{[1]Shopify Help Center. Editing robots.txt.liquid}

How it works

The standard is the Robots Exclusion Protocol, which Martijn Koster defined in 1994 and which the IETF published as RFC 9309 in 2022. The rules have to be in a file called /robots.txt, all lowercase, at the top level of the site.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} Google adds a nuance that is often forgotten: the rules only apply to the host, protocol and port where the file lives, so every subdomain needs its own.⁠^{[3]Google Crawling Infrastructure. How Google interprets the robots.txt specification}

The file is organized in groups. Each group starts with one or more User-agent lines, which name a bot, and continues with its Allow and Disallow rules. Each crawler looks for the group that names it, case-insensitively, and if none does it obeys the User-agent: * group.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} Google puts it more clearly: only one group applies to a crawler, the one with the most specific name that matches it, and the specific group and the general one are not combined.⁠^{[3]Google Crawling Infrastructure. How Google interprets the robots.txt specification} In other words: if you give a bot a group of its own, it stops reading the asterisk's rules.

Within a group, the most specific rule wins, the one that matches the most characters of the path, and if an Allow and a Disallow tie, the Allow applies.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} In Google's example, with allow: /p and disallow: /, the URL /page can be crawled, because the first rule is more specific. Paths are case-sensitive, and they support two wildcards: * for any sequence of characters and $ for the end of the URL.⁠^{[3]Google Crawling Infrastructure. How Google interprets the robots.txt specification}

Google only recognizes four fields: user-agent, allow, disallow and sitemap. Crawl-delay isn't among them and Google ignores it.⁠^{[3]Google Crawling Infrastructure. How Google interprets the robots.txt specification} Anthropic, on the other hand, says it supports that non-standard extension.⁠^{[4]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?} The Sitemap line points to your XML sitemap with a full URL and doesn't depend on any group: every crawler can follow it.⁠^{[3]Google Crawling Infrastructure. How Google interprets the robots.txt specification}

Three operational details weigh more than the syntax:

• The response code counts. If the file returns a 4xx, the crawler may request anything. If it returns a 5xx, the standard tells it to assume it may request nothing.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} Google, in that case, stops crawling the site for the first 12 hours while it keeps trying to read the file, and then uses the last saved copy for up to 30 days.⁠^{[3]Google Crawling Infrastructure. How Google interprets the robots.txt specification}
• Changes aren't immediate. The standard says a crawler shouldn't use the cached copy for more than 24 hours,⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} and OpenAI estimates about 24 hours for its search to adjust to a change.⁠^{[5]OpenAI. Overview of OpenAI Crawlers}
• Size has a limit. Google reads at most 500 KiB and ignores anything past that.⁠^{[3]Google Crawling Infrastructure. How Google interprets the robots.txt specification}

Finally, what the file isn't. The standard itself says these rules are not a form of access authorization and don't replace a security measure,⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} and Shopify describes them as guidance: not every crawler is obliged to follow them.⁠^{[1]Shopify Help Center. Editing robots.txt.liquid}

In a store

AI companies use several agents, and each one is allowed or blocked separately. OpenAI gives an example: you can let OAI-SearchBot through so you appear in ChatGPT search and block GPTBot to indicate that what it crawls shouldn't be used to train its models.⁠^{[5]OpenAI. Overview of OpenAI Crawlers} Google has a similar control, Google-Extended, which decides whether what it crawls can be used to train future versions of Gemini, and which, according to Google, doesn't affect your presence in Search or your ranking.⁠^{[6]Google Crawling Infrastructure. Google's common crawlers} The full table of agents is in the GEO entry.

Here is what the file might look like for a store that wants to keep the cart and sorted collections out, and not give up its content to train models. It is a fictional example:

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /collections/*sort_by*

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://example-store.com/sitemap.xml

What each part does. The first group applies to any bot that doesn't have its own: Googlebot, OAI-SearchBot and the rest can request the product pages, but not the cart or the reordered collections, which only multiply faceted navigation URLs. The second group names three training agents and closes the whole site to them. Notice what doesn't appear: OAI-SearchBot has no group of its own, on purpose. If you gave it one with an Allow: /, it would stop reading the asterisk and could request /cart and /checkout too.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}⁠^{[3]Google Crawling Infrastructure. How Google interprets the robots.txt specification}

On Shopify you don't start from zero. Every store has a default robots.txt that blocks, among other paths, the admin panel, the cart, checkout, the customer account, orders and sorted or filtered collections. To change it you create the robots.txt.liquid template in the theme, and Shopify warns that it is an unsupported customization. Blocking AI crawlers there only affects what is discovered on the open web: it doesn't stop the product data that Shopify Catalog sends to the agentic storefronts you have turned on.⁠^{[1]Shopify Help Center. Editing robots.txt.liquid}

On BigCommerce the file is edited at Settings › Website › Search Engine Robots. What you see there isn't everything: the platform applies separate rules to AI crawlers and high-volume ones, with a 10-second delay and sensitive pages blocked, which only show up if you open the published file. By default you don't need to add anything for AI bots to crawl the store.⁠^{[7]BigCommerce Support. Understanding the Robots.txt File}

How to check it

1. Open yourdomain.com/robots.txt and check that it answers with a 200. Do the same on every subdomain that serves pages, because each one carries its own rules.⁠^{[3]Google Crawling Infrastructure. How Google interprets the robots.txt specification}
2. For each bot you care about, look for the group that names it. If there isn't one, the asterisk's applies. Then see what that group says about the path of your product pages, for example /products/.
3. Open the robots.txt report in Search Console. It shows the files Google found on the top 20 hosts of your site, when it last read them and any errors or warnings. If you have fixed something urgent, you can ask it from there to read the file again.⁠^{[10]Search Console Help. robots.txt report}
4. Test a specific URL with URL Inspection, which tells you whether Google is blocked from accessing it.⁠^{[10]Search Console Help. robots.txt report}
5. On Shopify or BigCommerce, read the published file, not the settings screen: on BigCommerce, the rules for AI bots only show up there.⁠^{[7]BigCommerce Support. Understanding the Robots.txt File}

Related concept: noindex

robots.txt controls crawling; noindex controls indexing. The noindex rule goes in a <meta name="robots"> tag or in the X-Robots-Tag HTTP header, and when Googlebot reads it, Google drops the page from its results even if other sites link to it.⁠^{[9]Google Search Central. Block Search indexing with noindex} That is why they aren't combined lightly: if you block the URL in robots.txt, the crawler never gets to read the noindex and the page can keep appearing.⁠^{[9]Google Search Central. Block Search indexing with noindex} The right order for retiring a product page is noindex first with crawling open, and the block afterwards, if needed. When it makes sense to close the door to each AI bot is a different decision, and you'll find it in AI bot blocking.

To keep going, the guide to what an AI crawler sees on your store reads the file agent by agent, and the crawling and indexing reading path puts the other terms in order.