What an AI crawler sees in your store and how to check it

If you sell on Shopify or BigCommerce, you've probably read that AI bots don't run JavaScript, or that ChatGPT reads the JSON-LD on your product pages. It almost never comes with a source. This guide doesn't ask you to believe either claim. It shows you how to check what an AI crawler can read on your product page, with the rules as the standard and each company write them, in five steps that each end in something you can see.

What an AI crawler downloads

Before it requests a page, a well-behaved crawler requests your robots.txt and looks for the rules that apply to it.⁠^{[1]IETF (Koster, Illyes, Zeller and Sassman; September 2022). RFC 9309: Robots Exclusion Protocol} Then it requests the URL and gets whatever your server sends back: the HTML as it leaves the server, before anything runs. That is what any agent you let in downloads.

What JavaScript adds after the page loads depends on JavaScript rendering, and of the four companies whose agents this guide looks at, OpenAI, Perplexity, Anthropic and Google, only Google documents that it renders. Googlebot crawls the URL, queues the pages that return a 200 for rendering, renders them with headless Chromium, and what comes out goes on to indexing. Even so, Google says server-side rendering or prerendering is still a good idea, because it makes the site faster for people and crawlers and because not all bots can run JavaScript.⁠^{[2]Google Search Central. Understand the JavaScript SEO basics} OpenAI, Perplexity and Anthropic say nothing about JavaScript in their crawler documentation, which we read on October 5, 2026.⁠^{[3]OpenAI. Overview of OpenAI Crawlers}⁠^{[4]Perplexity. Perplexity Crawlers}⁠^{[5]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?} They don't say they don't run it. They just don't say.

Structured data works the same way as text. An application/ld+json block written into the server's HTML travels with every download, and Google uses the structured data it finds to understand the page's content and to show rich results.⁠^{[6]Google Search Central. Introduction to structured data markup in Google Search} If JavaScript generates the Product markup, Google warns that Shopping crawls may be less frequent and less reliable, which is a problem for data that changes fast, like price and availability.⁠^{[7]Google Search Central. Generate structured data with JavaScript}

For its AI features, Google doesn't ask for new files or special markup: the page has to be indexed, eligible to be shown with a snippet, and carry structured data that matches the visible text.⁠^{[8]Google Search Central. AI features and your website} Which agent searches and which one trains, company by company, is in the GEO entry. Here we stick to the practical part: how each one gets in and how you check it.

Step 1: read your robots.txt the way each agent reads it

Open your robots.txt in the browser, at the root of your domain. If you came here looking for what to put in robots.txt for ChatGPT, the first surprise is that ChatGPT isn't one agent but three, and each one reads the file on its own. The rules, as the standard (RFC 9309) and Google write them:

• Each agent obeys one group only: the one that names it. It looks for its name without regard to case and, if it finds it, stops reading the User-agent: * group. It only falls back to * if no group names it.⁠^{[1]IETF (Koster, Illyes, Zeller and Sassman; September 2022). RFC 9309: Robots Exclusion Protocol}⁠^{[9]Google Search Central. How Google interprets the robots.txt specification}
• Within the group, the rule with the longest matching path wins, and if an Allow and a Disallow tie, the Allow wins. Google calls this using the least restrictive rule.⁠^{[1]IETF (Koster, Illyes, Zeller and Sassman; September 2022). RFC 9309: Robots Exclusion Protocol}⁠^{[9]Google Search Central. How Google interprets the robots.txt specification}
• If the file returns a 4xx error, the agent may crawl everything. On a 5xx, the standard says to assume nothing can be crawled; Google stops for 12 hours, keeps using its cached copy for up to 30 days and, if the error continues, acts as if there were no restrictions.⁠^{[1]IETF (Koster, Illyes, Zeller and Sassman; September 2022). RFC 9309: Robots Exclusion Protocol}⁠^{[9]Google Search Central. How Google interprets the robots.txt specification}
• The file is cached for up to 24 hours, and OpenAI estimates about 24 hours for its systems to pick up a change.⁠^{[1]IETF (Koster, Illyes, Zeller and Sassman; September 2022). RFC 9309: Robots Exclusion Protocol}⁠^{[9]Google Search Central. How Google interprets the robots.txt specification}⁠^{[3]OpenAI. Overview of OpenAI Crawlers}
• Crawl-delay isn't part of the standard: Anthropic supports it, Google doesn't, and OpenAI and Perplexity say nothing.⁠^{[5]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?}⁠^{[9]Google Search Central. How Google interprets the robots.txt specification}
• And what the file is not: the standard itself says these rules are not a form of access authorization.⁠^{[1]IETF (Koster, Illyes, Zeller and Sassman; September 2022). RFC 9309: Robots Exclusion Protocol}

These are the names the four companies' agents look for in the file, what each company says about whether they follow it, and where it publishes their IP addresses:

Google-Extended isn't in the table because it isn't an agent: it has no user agent of its own. Google crawls with its usual agents and uses that name in robots.txt only as a control. According to Google, it doesn't affect your inclusion in Search and isn't a ranking signal.⁠^{[12]Google Search Central. Google's common crawlers}

The first rule has a practical consequence: give an agent a group of its own and you take away the general group's rules. In this fictitious example, OAI-SearchBot finds its group, stops reading *, and is now allowed into /cart and /checkout. If you want a group for one agent, copy the general group's Disallow lines into it.

User-agent: *
Disallow: /cart
Disallow: /checkout

User-agent: OAI-SearchBot
Allow: /

Shopify. The default file blocks /admin, /cart, /checkout, /checkouts/, /account, /orders and sorted and filtered collections. To change it, you edit the theme's robots.txt.liquid template, and Shopify warns that this is an unsupported customization and that getting it wrong can cost you all your traffic.⁠^{[13]Shopify Help Center. Editing robots.txt.liquid} Its developer documentation recommends starting from the Liquid objects that reproduce the default rules (robots.default_groups) rather than rewriting the file, and its example for allowing a crawler is a group with Allow: / and nothing else: copied as-is for an agent, it leaves that agent without the /cart and /checkout rules.⁠^{[14]Shopify.dev. Customize robots.txt} Shopify also handles bots at the network layer without you doing anything, and warns that robots.txt rules are advisory: not every crawler follows them. Blocking AI crawlers there or at the network level only affects what gets discovered on the open web, and doesn't stop the product data that Shopify Catalog sends to any agentic storefronts you have turned on.⁠^{[13]Shopify Help Center. Editing robots.txt.liquid}

BigCommerce. You edit the file under Settings › Website › Search Engine Robots, and with Multi-Storefront, per storefront. But what you see there isn't the whole file: BigCommerce applies a separate group to AI and high-volume crawlers, with Crawl-delay: 10 and a few blocked paths, which you can't see or edit in the settings but which does show up in your /robots.txt. By default, you don't need to add anything for AI bots to crawl your store.⁠^{[15]BigCommerce Support. Understanding the Robots.txt File}

On both platforms the rule is the same: read the live file, not the settings screen. For each agent in the table, find the group that names it, or whether it falls into *, and what that group says about the path to your product pages. When it makes sense to keep each one out is a separate decision, covered in when to block each agent.

Step 2: look at the HTML your server sends

Open a product page and view its source: "View page source" in the right-click menu, or view-source: in front of the URL. Don't use the inspector in your browser's developer tools, which shows the page after JavaScript has run, not what left the server. In that source, look for every piece of data you want an agent to read:

• the product name, the price and the availability;
• the description, the reviews and the FAQs;
• application/ld+json: how many blocks there are and which @type each one declares. On a product page, you'd expect a Product.

Whatever you find in the source arrives with any download. Anything you see on the page that doesn't appear here was put there by JavaScript, and only an agent that renders will read it. If your theme builds the product page with server-side rendering, that data will be in the source; whatever an app or a script adds after the page loads won't be. To save a copy to compare in the next steps, one line in the terminal does it: curl -s -o server.html https://your-store.example/products/example (the URL is fictitious).

Step 3: compare it with what Google renders

The only renderer you can look through is Google's, and it has two windows:

• The URL Inspection tool in Search Console. "View crawled page" shows the HTML your server returned, the HTTP headers, the JavaScript console messages and the resources the page loaded. "Test live URL" checks the URL right now and, if the test succeeds and the URL is public, adds a screenshot of what it rendered. The "Crawled as" field tells you whether it came in as mobile or desktop.⁠^{[16]Search Console Help. URL Inspection tool}
• The Rich Results Test. Run it with the URL, not by pasting code: code mode has JavaScript limits.⁠^{[7]Google Search Central. Generate structured data with JavaScript} It crawls as a smartphone unless you choose otherwise, and it lists the structured data types it finds.⁠^{[17]Search Console Help. Rich Results Test} If your Product shows up here and wasn't in the source from step 2, JavaScript generates it: Google sees it, with the Shopping caveat from the first section, and an agent that doesn't render doesn't.

It's Google's renderer, and it only tells you what Google sees. For every other agent, what counts is the source from step 2. If what you want is to check the markup itself, there's a guide to auditing a product page's structured data, and it's worth remembering that validating markup is not the same as verifying it's published.

Step 4: request the page with each agent's user agent

If you were looking for ChatGPT's user agent, there are three, one per agent. Each company publishes its full strings: OAI-SearchBot/1.4, GPTBot/1.4 and ChatGPT-User/1.0 for OpenAI;⁠^{[3]OpenAI. Overview of OpenAI Crawlers} PerplexityBot/1.0 and Perplexity-User/1.0 for Perplexity;⁠^{[4]Perplexity. Perplexity Crawlers} Googlebot/2.1, in mobile and desktop versions, for Google.⁠^{[12]Google Search Central. Google's common crawlers} Anthropic publishes its agents' names but not the full strings.⁠^{[5]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?} When you filter, search for the name, not the whole string, because the version changes.

Request the same product page with your browser and with each agent's full string, and compare the responses: the status code, the size, and whether you get the product page or a security check page. In the terminal it's one line per agent: curl -A "<agent string>" -o agent.html <your product page URL>.

This tells you whether your server, your CDN or your firewall changes the response based on the user agent. It doesn't tell you what the real crawler gets, because a borrowed user agent isn't the crawler: your request comes from your IP, and the real agents come from the ranges each company publishes. Perplexity advises anyone configuring a firewall to combine the string with an IP check,⁠^{[4]Perplexity. Perplexity Crawlers} and a firewall that does that may treat your request as what it is: an imitation. That's why Google asks that crawling be allowed in robots.txt and also in any CDN or hosting infrastructure:⁠^{[8]Google Search Central. AI features and your website} an Allow in the file doesn't open the firewall.

Step 5: confirm real visits in the logs

The real crawler only shows up in the request logs of your server or your CDN. Filter by the agent's name and check the IP on each line:

• OpenAI, Perplexity and Anthropic publish their ranges in the JSON lists from the table in step 1.⁠^{[3]OpenAI. Overview of OpenAI Crawlers}⁠^{[4]Perplexity. Perplexity Crawlers}⁠^{[5]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?}
• Google is verified with a reverse DNS lookup of the IP, which has to return a googlebot.com, google.com or googleusercontent.com domain, followed by a forward lookup on that domain; or with its JSON lists.⁠^{[11]Google Search Central. Verifying Googlebot and other Google crawlers}

A sample line, fictitious (the IP is from a range reserved for examples):

203.0.113.24 - - [05/Oct/2026:10:14:07 +0200] "GET /products/example HTTP/1.1" 200 48213 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

That line says the request calls itself GPTBot, and it isn't GPTBot until its IP is in gptbot.json. In the logs, look at:

• how often it requests robots.txt;
• the status codes: a 200 is the page, a 403 is a block and a 429 means your server has asked it to slow down;
• whether it requests the JavaScript, CSS or images after the HTML, which is a sign that it renders;
• the bytes: a response much smaller than your product page may be a challenge page.

The lists change, and blocking by IP has its limits: Anthropic warns that blocking its addresses may not work well or guarantee permanent exclusion.⁠^{[5]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?} If your store runs on a hosted platform, ask what logs you get before planning this check. Shopify, for example, doesn't recommend putting a proxy in front of your store.⁠^{[13]Shopify Help Center. Editing robots.txt.liquid}

What we tested

What we checked from the outside, on October 5, 2026

From one of our own computers, without being any crawler, we read public files and requested product pages with borrowed strings. The stores go unnamed, except for BigCommerce's demo store, which belongs to the platform itself.

• Shopify's default file. On two large Shopify stores, the live robots.txt only has groups for * and for a few ad and SEO tool crawlers. It names none of the agents in the table, so they all fall into * and its rules.
• BigCommerce's separate group. The live file of its demo store, cornerstone-light-demo.mybigcommerce.com, has a third group that names OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot and Google-Extended, among several dozen agents, with Crawl-delay: 10 and a few blocked paths. Claude-SearchBot, Claude-User and Perplexity-User aren't in it, so * governs them.
• The same HTML with any string. We requested a product page from three Shopify stores with a browser's string and with those of OAI-SearchBot, GPTBot, PerplexityBot and Googlebot. Each store returned a 200 and the same HTML, byte for byte, with two application/ld+json blocks, one of them with a Product node. It's our IP with a borrowed string: it says those servers don't change the response based on the user agent, not what any crawler receives.
• A wall instead of the file. When we requested another large store's robots.txt, we got a 429 and a security check page. That, too, is what an agent may get.

The test with the real agents

• Setup. A public URL built as a product page, labeled "fictitious example" unless it's a client's product page used with their permission (in which case the page adds real-example). No links to it, left out of the sitemap, and with a robots.txt that lets in every agent in the table; the robots.txt is saved as it was served that day. A password-protected Shopify development store won't do. On it, four canary values, each a random eight-character string with no results on the web: C1, visible text in the server's HTML; C2, only inside the Product node of the server's JSON-LD (for example, in mpn), not visible on the page; C3, visible text that JavaScript writes after the page loads and that isn't in the server's HTML; C4, a JSON-LD property that only JavaScript adds. Check in the source that C1 and C2 are there and C3 and C4 aren't, and in the browser that C1 and C3 are visible. Note the hosting, the platform and its version, the theme and its version, and the date.
• Runs. ChatGPT with search, Perplexity and Claude with web access: three runs on each, each one in a new chat, with the session state noted and the start time to the minute, to cross-check with the logs. A single query in English, written before starting and run from the United States, that asks for the values labeled "copied verbatim" from the URL without revealing them. Plus Google's renderer: the Rich Results Test with the URL (does it detect C2? and C4?) and, if the property is verified, "Test live URL" in URL Inspection (does C3 appear?). Gemini and Copilot are optional, only if the reviewer wants them: their agents aren't in the sources we've opened, so whatever agent shows up in the logs is noted without interpreting it.
• Logs. For each run window: the time, the IP, the user agent, the path, the status and the bytes. For each request: the agent; whether the IP is on its company's list, with the list's creation date; whether it requested robots.txt first; whether it requested JavaScript, CSS or images after the HTML; and the status code. If a real store's logs go back thirty days, an optional second table with requests per agent, verified and unverified.
• What gets published. Where each canary value was (and the strings themselves, if the reviewer approves), the table below with its date, a line with the store and its versions, and the limits.

Tested on: hosting, platform and version, theme and version, date.

The limits, plainly: one page, one day and three runs per assistant. ChatGPT-User, Perplexity-User and Claude-User, the agents a person triggers, aren't the search crawlers; those only show up if they happen to pass through within the log window. What an assistant answers is a model's output: the random values rule out a lucky guess, not that it holds something back, so a "no" weighs less than a "yes". And agent behavior and the IP lists both change.