# What an AI crawler sees in your store and how to check it > What an AI crawler like GPTBot or OAI-SearchBot sees in your store: your robots.txt, the HTML your server sends, and how to check it step by step. URL: https://clione.ai/guides/what-an-ai-crawler-sees/ Lang: en If you sell on Shopify or BigCommerce, you've probably read that AI bots don't run JavaScript, or that ChatGPT reads the [JSON-LD](/glossary/json-ld/) on your product pages. It almost never comes with a source. This guide doesn't ask you to believe either claim. It shows you how to check what an [AI crawler](/glossary/web-crawler/) can read on your [product page](/glossary/product-page/), with the rules as the standard and each company write them, in five steps that each end in something you can see. ## What an AI crawler downloads Before it requests a page, a well-behaved crawler requests your [robots.txt](/glossary/robots-txt/) and looks for the rules that apply to it.[^rfc9309] Then it requests the URL and gets whatever your server sends back: the HTML as it leaves the server, before anything runs. **That is what any agent you let in downloads.** What JavaScript adds after the page loads depends on [JavaScript rendering](/glossary/javascript-rendering/), and of the four companies whose agents this guide looks at, OpenAI, [Perplexity](/glossary/perplexity/), Anthropic and Google, only Google documents that it renders. [Googlebot](/glossary/googlebot/) crawls the URL, queues the pages that return a 200 for rendering, renders them with headless Chromium, and what comes out goes on to [indexing](/glossary/indexing/). Even so, Google says [server-side rendering](/glossary/server-side-rendering/) or prerendering is still a good idea, because it makes the site faster for people and crawlers and because **not all bots can run JavaScript**.[^google-js-seo] OpenAI, Perplexity and Anthropic say nothing about JavaScript in their crawler documentation, which we read on October 5, 2026.[^openai-bots][^perplexity-bots][^anthropic-bots] They don't say they don't run it. They just don't say. [Structured data](/glossary/structured-data/) works the same way as text. An `application/ld+json` block written into the server's HTML travels with every download, and Google uses the structured data it finds to understand the page's content and to show [rich results](/glossary/rich-results/).[^google-structured-data] If JavaScript generates the [Product](/glossary/product-schema/) markup, Google warns that Shopping crawls may be less frequent and less reliable, which is a problem for data that changes fast, like [price and availability](/glossary/price-and-availability/).[^google-js-sd] For its AI features, Google doesn't ask for new files or special markup: the page has to be indexed, eligible to be shown with a snippet, and carry structured data that matches the visible text.[^google-ai-features] Which agent searches and which one trains, company by company, is in the [GEO entry](/glossary/geo/). Here we stick to the practical part: how each one gets in and how you check it. ## Step 1: read your robots.txt the way each agent reads it Open your `robots.txt` in the browser, at the root of your domain. If you came here looking for what to put in `robots.txt` for ChatGPT, the first surprise is that ChatGPT isn't one agent but three, and each one reads the file on its own. The rules, as the standard (RFC 9309) and Google write them: - **Each agent obeys one group only**: the one that names it. It looks for its name without regard to case and, if it finds it, stops reading the `User-agent: *` group. It only falls back to `*` if no group names it.[^rfc9309][^google-robots-spec] - Within the group, the rule with the longest matching path wins, and if an `Allow` and a `Disallow` tie, the `Allow` wins. Google calls this using the least restrictive rule.[^rfc9309][^google-robots-spec] - If the file returns a 4xx error, the agent may crawl everything. On a 5xx, the standard says to assume nothing can be crawled; Google stops for 12 hours, keeps using its cached copy for up to 30 days and, if the error continues, acts as if there were no restrictions.[^rfc9309][^google-robots-spec] - The file is cached for up to 24 hours, and OpenAI estimates about 24 hours for its systems to pick up a change.[^rfc9309][^google-robots-spec][^openai-bots] - `Crawl-delay` isn't part of the standard: Anthropic supports it, Google doesn't, and OpenAI and Perplexity say nothing.[^anthropic-bots][^google-robots-spec] - And what the file is not: the standard itself says these rules are not a form of access authorization.[^rfc9309] These are the names the four companies' agents look for in the file, what each company says about whether they follow it, and where it publishes their IP addresses: | Company | Agent (the name that goes in `robots.txt`) | Follows `robots.txt`? Per its documentation | IP list | | ---------- | ------------------------------------------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | OpenAI | [OAI-SearchBot](/glossary/oai-searchbot/) | Controlled in `robots.txt`, separately from the other two.[^openai-bots] | [`openai.com/searchbot.json`](https://openai.com/searchbot.json) | | OpenAI | [GPTBot](/glossary/gptbot/) | Controlled in `robots.txt`, separately from the other two.[^openai-bots] | [`openai.com/gptbot.json`](https://openai.com/gptbot.json) | | OpenAI | [ChatGPT-User](/glossary/chatgpt-user/) | Maybe not: a person triggers it from ChatGPT.[^openai-bots] | [`openai.com/chatgpt-user.json`](https://openai.com/chatgpt-user.json) | | Perplexity | [PerplexityBot](/glossary/perplexitybot/) | Yes.[^perplexity-bots] | [`perplexity.com/perplexitybot.json`](https://www.perplexity.com/perplexitybot.json) | | Perplexity | Perplexity-User | Generally ignores it: a person triggers it.[^perplexity-bots] | [`perplexity.com/perplexity-user.json`](https://www.perplexity.com/perplexity-user.json) | | Anthropic | [ClaudeBot](/glossary/claudebot/), Claude-SearchBot and Claude-User | Yes, and supports `Crawl-delay`.[^anthropic-bots] | [`claude.com/crawling/bots.json`](https://claude.com/crawling/bots.json) | | Google | Googlebot | Yes, for automatic crawls.[^google-crawlers-overview] | By reverse DNS or with its JSON lists.[^google-verify] | [Google-Extended](/glossary/google-extended/) isn't in the table because it isn't an agent: it has no [user agent](/glossary/user-agent/) of its own. Google crawls with its usual agents and uses that name in `robots.txt` only as a control. According to Google, it doesn't affect your inclusion in Search and isn't a ranking signal.[^google-common-crawlers] The first rule has a practical consequence: give an agent a group of its own and you take away the general group's rules. In this fictitious example, OAI-SearchBot finds its group, stops reading `*`, and is now allowed into `/cart` and `/checkout`. If you want a group for one agent, copy the general group's `Disallow` lines into it. ```text User-agent: * Disallow: /cart Disallow: /checkout User-agent: OAI-SearchBot Allow: / ``` **Shopify.** The default file blocks `/admin`, `/cart`, `/checkout`, `/checkouts/`, `/account`, `/orders` and sorted and filtered collections. To change it, you edit the theme's `robots.txt.liquid` template, and Shopify warns that this is an unsupported customization and that getting it wrong can cost you all your traffic.[^shopify-robots] Its developer documentation recommends starting from the [Liquid](/glossary/shopify-liquid/) objects that reproduce the default rules (`robots.default_groups`) rather than rewriting the file, and its example for allowing a crawler is a group with `Allow: /` and nothing else: copied as-is for an agent, it leaves that agent without the `/cart` and `/checkout` rules.[^shopify-robots-liquid] Shopify also handles bots at the network layer without you doing anything, and warns that `robots.txt` rules are advisory: not every crawler follows them. Blocking AI crawlers there or at the network level only affects what gets discovered on the open web, and doesn't stop the product data that [Shopify Catalog](/glossary/shopify-catalog/) sends to any [agentic storefronts](/glossary/agentic-storefronts/) you have turned on.[^shopify-robots] **BigCommerce.** You edit the file under `Settings › Website › Search Engine Robots`, and with Multi-Storefront, per storefront. But what you see there isn't the whole file: BigCommerce applies a separate group to AI and high-volume crawlers, with `Crawl-delay: 10` and a few blocked paths, which you can't see or edit in the settings but which does show up in your `/robots.txt`. By default, you don't need to add anything for AI bots to crawl your store.[^bigcommerce-robots] On both platforms the rule is the same: **read the live file, not the settings screen**. For each agent in the table, find the group that names it, or whether it falls into `*`, and what that group says about the path to your product pages. When it makes sense to keep each one out is a separate decision, covered in [when to block each agent](/glossary/ai-bot-blocking/). ## Step 2: look at the HTML your server sends Open a product page and view its source: "View page source" in the right-click menu, or `view-source:` in front of the URL. Don't use the inspector in your browser's developer tools, which shows the page after JavaScript has run, not what left the server. In that source, look for every piece of data you want an agent to read: - the product name, the price and the availability; - the description, the reviews and the FAQs; - `application/ld+json`: how many blocks there are and which `@type` each one declares. On a product page, you'd expect a Product. **Whatever you find in the source arrives with any download.** Anything you see on the page that doesn't appear here was put there by JavaScript, and only an agent that renders will read it. If your theme builds the product page with [server-side rendering](/glossary/server-side-rendering/), that data will be in the source; whatever an app or a script adds after the page loads won't be. To save a copy to compare in the next steps, one line in the terminal does it: `curl -s -o server.html https://your-store.example/products/example` (the URL is fictitious). ## Step 3: compare it with what Google renders The only renderer you can look through is Google's, and it has two windows: - **The URL Inspection tool in [Search Console](/glossary/google-search-console/).** "View crawled page" shows the HTML your server returned, the HTTP headers, the JavaScript console messages and the resources the page loaded. "Test live URL" checks the URL right now and, if the test succeeds and the URL is public, adds a screenshot of what it rendered. The "Crawled as" field tells you whether it came in as mobile or desktop.[^gsc-url-inspection] - **The [Rich Results Test](/glossary/rich-results-test/).** Run it with the URL, not by pasting code: code mode has JavaScript limits.[^google-js-sd] It crawls as a smartphone unless you choose otherwise, and it lists the structured data types it finds.[^google-rrt] If your Product shows up here and wasn't in the source from step 2, JavaScript generates it: Google sees it, with the Shopping caveat from the first section, and an agent that doesn't render doesn't. **It's Google's renderer, and it only tells you what Google sees.** For every other agent, what counts is the source from step 2. If what you want is to check the markup itself, there's a guide to [auditing a product page's structured data](/guides/audit-product-page-structured-data/), and it's worth remembering that [validating markup is not the same as verifying it's published](/comparisons/validate-vs-verify/). ## Step 4: request the page with each agent's user agent If you were looking for ChatGPT's user agent, there are three, one per agent. Each company publishes its full strings: OAI-SearchBot/1.4, GPTBot/1.4 and ChatGPT-User/1.0 for OpenAI;[^openai-bots] PerplexityBot/1.0 and Perplexity-User/1.0 for Perplexity;[^perplexity-bots] Googlebot/2.1, in mobile and desktop versions, for Google.[^google-common-crawlers] Anthropic publishes its agents' names but not the full strings.[^anthropic-bots] When you filter, search for the name, not the whole string, because the version changes. Request the same product page with your browser and with each agent's full string, and compare the responses: the status code, the size, and whether you get the product page or a security check page. In the terminal it's one line per agent: `curl -A "" -o agent.html `. This tells you whether your server, your [CDN](/glossary/cdn-caching-for-bots/) or your firewall changes the response based on the user agent. It doesn't tell you what the real crawler gets, because **a borrowed user agent isn't the crawler**: your request comes from your IP, and the real agents come from the ranges each company publishes. Perplexity advises anyone configuring a firewall to combine the string with an IP check,[^perplexity-bots] and a firewall that does that may treat your request as what it is: an imitation. That's why Google asks that crawling be allowed in `robots.txt` and also in any CDN or hosting infrastructure:[^google-ai-features] an `Allow` in the file doesn't open the firewall. ## Step 5: confirm real visits in the logs The real crawler only shows up in the request logs of your server or your CDN. Filter by the agent's name and check the IP on each line: - OpenAI, Perplexity and Anthropic publish their ranges in the JSON lists from the table in step 1.[^openai-bots][^perplexity-bots][^anthropic-bots] - Google is verified with a reverse DNS lookup of the IP, which has to return a `googlebot.com`, `google.com` or `googleusercontent.com` domain, followed by a forward lookup on that domain; or with its JSON lists.[^google-verify] A sample line, fictitious (the IP is from a range reserved for examples): ```text 203.0.113.24 - - [05/Oct/2026:10:14:07 +0200] "GET /products/example HTTP/1.1" 200 48213 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" ``` That line says the request calls itself GPTBot, and **it isn't GPTBot until its IP is in `gptbot.json`**. In the logs, look at: - how often it requests `robots.txt`; - the status codes: a 200 is the page, a 403 is a block and a 429 means your server has asked it to slow down; - whether it requests the JavaScript, CSS or images after the HTML, which is a sign that it renders; - the bytes: a response much smaller than your product page may be a challenge page. The lists change, and blocking by IP has its limits: Anthropic warns that blocking its addresses may not work well or guarantee permanent exclusion.[^anthropic-bots] If your store runs on a hosted platform, ask what logs you get before planning this check. Shopify, for example, doesn't recommend putting a proxy in front of your store.[^shopify-robots] ## What we tested ### What we checked from the outside, on October 5, 2026 From one of our own computers, without being any crawler, we read public files and requested product pages with borrowed strings. The stores go unnamed, except for BigCommerce's demo store, which belongs to the platform itself. - **Shopify's default file.** On two large Shopify stores, the live `robots.txt` only has groups for `*` and for a few ad and SEO tool crawlers. It names none of the agents in the table, so they all fall into `*` and its rules. - **BigCommerce's separate group.** The live file of its demo store, `cornerstone-light-demo.mybigcommerce.com`, has a third group that names OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot and Google-Extended, among several dozen agents, with `Crawl-delay: 10` and a few blocked paths. Claude-SearchBot, Claude-User and Perplexity-User aren't in it, so `*` governs them. - **The same HTML with any string.** We requested a product page from three Shopify stores with a browser's string and with those of OAI-SearchBot, GPTBot, PerplexityBot and Googlebot. Each store returned a 200 and the same HTML, byte for byte, with two `application/ld+json` blocks, one of them with a Product node. It's our IP with a borrowed string: it says those servers don't change the response based on the user agent, not what any crawler receives. - **A wall instead of the file.** When we requested another large store's `robots.txt`, we got a 429 and a security check page. That, too, is what an agent may get. ### The test with the real agents - **Setup.** A public URL built as a product page, labeled "fictitious example" unless it's a client's product page used with their permission (in which case the page adds `real-example`). No links to it, left out of the sitemap, and with a `robots.txt` that lets in every agent in the table; the `robots.txt` is saved as it was served that day. A password-protected Shopify development store won't do. On it, four canary values, each a random eight-character string with no results on the web: C1, visible text in the server's HTML; C2, only inside the Product node of the server's JSON-LD (for example, in `mpn`), not visible on the page; C3, visible text that JavaScript writes after the page loads and that isn't in the server's HTML; C4, a JSON-LD property that only JavaScript adds. Check in the source that C1 and C2 are there and C3 and C4 aren't, and in the browser that C1 and C3 are visible. Note the hosting, the platform and its version, the theme and its version, and the date. - **Runs.** ChatGPT with search, Perplexity and Claude with web access: three runs on each, each one in a new chat, with the session state noted and the start time to the minute, to cross-check with the logs. A single query in English, written before starting and run from the United States, that asks for the values labeled "copied verbatim" from the URL without revealing them. Plus Google's renderer: the Rich Results Test with the URL (does it detect C2? and C4?) and, if the property is verified, "Test live URL" in URL Inspection (does C3 appear?). [Gemini](/glossary/gemini/) and [Copilot](/glossary/copilot/) are optional, only if the reviewer wants them: their agents aren't in the sources we've opened, so whatever agent shows up in the logs is noted without interpreting it. - **Logs.** For each run window: the time, the IP, the user agent, the path, the status and the bytes. For each request: the agent; whether the IP is on its company's list, with the list's creation date; whether it requested `robots.txt` first; whether it requested JavaScript, CSS or images after the HTML; and the status code. If a real store's logs go back thirty days, an optional second table with requests per agent, verified and unverified. - **What gets published.** Where each canary value was (and the strings themselves, if the reviewer approves), the table below with its date, a line with the store and its versions, and the limits. Tested on: hosting, platform and version, theme and version, date. | Assistant | Run | Agent in the log | IP verified | Requested `robots.txt` | Requested JS, CSS or images | Status | C1 · C2 · C3 · C4 | | ------------------------- | ------------ | ---------------- | ----------- | ---------------------- | --------------------------- | ------ | ----------------- | | ChatGPT with search | 1 | | | | | | | | ChatGPT with search | 2 | | | | | | | | ChatGPT with search | 3 | | | | | | | | Perplexity | 1 | | | | | | | | Perplexity | 2 | | | | | | | | Perplexity | 3 | | | | | | | | Claude with web access | 1 | | | | | | | | Claude with web access | 2 | | | | | | | | Claude with web access | 3 | | | | | | | | Google: Rich Results Test | With the URL | | | | | | | | Google: URL Inspection | Live test | | | | | | | The limits, plainly: one page, one day and three runs per assistant. ChatGPT-User, Perplexity-User and Claude-User, the agents a person triggers, aren't the search crawlers; those only show up if they happen to pass through within the log window. What an assistant answers is a model's output: the random values rule out a lucky guess, not that it holds something back, so a "no" weighs less than a "yes". And agent behavior and the IP lists both change.