In beta, with limited placesOpen to everyone in Q1 2027

Help Log in

↑↓ to move · Enter to open · Esc to close

Reading path

Crawling and indexing for your store

For your store to show up in a search engine or in an AI answer, a program has to read it first. This path puts crawling and indexing in order, start to finish. By the end you will know who visits your store and how they identify themselves, what you tell them with robots.txt and what you cannot tell them with it, why a page can be read and not stored, and what to decide about the AI companies' bots, so you can judge any rule someone proposes you add or remove.

First who reads and what they ask for, then what you tell them about each URL, then who each bot is, and last what you publish with agents in mind. By the time you reach the AI bots, you will know what a file can and cannot ask of them. The path has five stretches.

  1. Steps 1 to 6 · How a search engine reads the web. The web crawler, indexing and the difference between the two. Then the two basic tools of the site: robots.txt, with the user agent each bot identifies itself with, and the XML sitemap.
  2. Steps 7 to 14 · Which page counts and how many get visited. The signals you use to choose what gets indexed and under which URL: noindex, the canonical URL, duplicate content, the 301 redirect, hreflang and internal linking. Two terms for large stores close the stretch: faceted navigation, which multiplies URLs, and crawl budget.
  3. Steps 15 to 23 · Who crawls your store. The search engines' crawlers, Googlebot and Bingbot, and the AI companies' crawlers, each with its own job: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and ClaudeBot. Google's two controls for its AI close the stretch: Google-Extended and the generative AI control in Search Console.
  4. Steps 24 to 28 · Opening or closing the door. AI bot blocking and what the network layer decides: caching and CDNs, bot signing with Web Bot Auth and charging per visit with pay per crawl. IndexNow closes it, the notice that a URL has changed.
  5. Steps 29 to 35 · For agents, and one measure apart. The files and responses offered to AI: llms.txt, agents.md, Markdown for agents, the content negotiation that serves it, the Link header and the .well-known directory. The last one, Core Web Vitals, is not about who gets in but about how the page loads, and it stays here because it is the one most often confused with crawling.

This is one of the seven reading paths. Read it after the path on AI in search engines if you do not yet know why the AI companies' bots visit you.

Start here

If you are starting from zero, read these three stops first: web crawler, crawling vs indexing and robots.txt. Then go back to the full path wherever you left off.

1

Web crawler

The program that downloads your pages and identifies itself by name: the lead of this whole path.

2

Crawling vs indexing

The difference between reading a page and storing it, which explains why a page can be crawled and not indexed.

4

robots.txt

The file where you tell each crawler which URLs it may request, and what it cannot do for you.

The path, step by step

  1. Web crawler

    A web crawler is a program that travels the web automatically: it downloads pages, follows their links to discover others and hands what it finds to the system it works for, such as a search engine's index or an AI assistant.

  2. Crawling vs indexing

    Crawling vs indexing is the difference between reading a page and storing it.

  3. Indexing

    Indexing is the step in which a search engine analyzes a page it has already crawled (its text, its tags, its images and the HTML that results from running its JavaScript), decides whether to keep it and under which canonical URL, and stores it in its index.

  4. robots.txt

    robots.txt is a text file placed at the root of a website that tells crawlers which URLs they may request and which they may not, with rules grouped by each bot's name.

  5. User agent

    The user agent is the text string with which a browser, a crawler or any other program identifies itself to a web server on every request, in the User-Agent HTTP header.

  6. XML sitemap

    An XML sitemap is a file in XML format that lists the URLs of a website that its owner wants search engines to know about, with optional data such as each one's last modification date.

  7. noindex

    noindex is a rule placed on a web page, as a robots meta tag in the HTML or as an X-Robots-Tag HTTP header, that asks search engines not to include it in their index or show it in their results.

  8. Canonical URL

    A canonical URL is the address a site declares as the preferred one when the same content is served from several URLs, with a rel=canonical annotation in a page's HTML or in its HTTP header.

  9. Duplicate content

    Duplicate content is identical or nearly identical content that can be opened from more than one URL, within one site or across different sites.

  10. 301 redirect

    A 301 redirect is the server response that says a URL has moved permanently to another address, given in the Location header.

  11. hreflang

    hreflang is the attribute a page uses to tell search engines which other URLs are versions of the same content in another language or for another region, with a code such as es, en-GB or x-default.

  12. Internal linking

    Internal linking is the set of links that connect the pages of a single website to each other: the menu, categories, breadcrumbs or links inside a text.

  13. Faceted navigation

    Faceted navigation is the filter system on a category or results page that lets shoppers combine attributes, such as size, color, brand or price, to narrow the products shown.

  14. Crawl budget

    Crawl budget is the set of URLs on a site that a search engine can and wants to crawl.

  15. Googlebot

    Googlebot is the generic name for the two web crawlers Google Search uses, one that simulates a mobile device and one a desktop computer, to download the pages that can then be indexed.

  16. Bingbot

    Bingbot is Microsoft Bing's standard web crawler: the program that downloads a site's pages so Bing can index them and use them in its results, in Copilot and in the answers that rely on its index.

  17. GPTBot

    GPTBot is OpenAI's web crawler, which collects content that may be used to train its generative AI foundation models.

  18. OAI-SearchBot

    OAI-SearchBot is OpenAI's web crawler for showing sites in the results of ChatGPT's search features.

  19. ChatGPT-User

    ChatGPT-User is the OpenAI web agent that visits a page to carry out an action a person requests in ChatGPT or in a custom GPT.

  20. PerplexityBot

    PerplexityBot is Perplexity's web crawler, which visits sites to show and link them in Perplexity's search results.

  21. ClaudeBot

    ClaudeBot is Anthropic's web crawler.

  22. Google-Extended

    Google-Extended is a name used in robots.txt to decide whether the content Google crawls on a site can help train future generations of the Gemini models and ground the answers of the Gemini apps and Vertex AI.

  23. Search Console generative AI control

    The Search Console generative AI control is the Google setting that lets a site owner decide whether their links and content can appear in Search's generative AI features: AI Overviews, AI Mode and Discover's generative AI.

  24. AI bot blocking

    AI bot blocking is the set of measures a website uses to stop or limit the agents of AI companies from requesting its pages, whether to train models, to search or to serve a user.

  25. Caching and CDNs for bots

    Caching and CDNs for bots are the HTTP layers that store copies of a store's pages, on the server or on a content delivery network (CDN), and answer crawlers and AI agents with them without generating the pages again.

  26. Web Bot Auth

    Web Bot Auth is a protocol, in IETF draft, with which a bot or an AI agent cryptographically signs its HTTP requests so that the site receiving them can check who sent them, without trusting the name it declares or its IP.

  27. Pay per crawl

    Pay per crawl is a Cloudflare feature, in closed beta since July 2025, that lets a site owner charge AI crawlers each time they request a page.

  28. IndexNow

    IndexNow is a public protocol a website uses to tell search engines, with an HTTP request, that a URL has been created, updated or deleted, instead of waiting for them to discover it by crawling.

  29. llms.txt

    The llms.txt is a Markdown text file, usually placed at the root of a website, that summarizes what the site is and links its most useful pages so that a language model or an AI agent can find them when it needs to answer.

  30. agents.md

    The agents.md file is a Markdown text file written for AI agents, not for people.

  31. Markdown for agents

    Markdown for agents is the practice of offering, alongside a page's HTML version, a version of the same content in Markdown for AI agents and language models to read.

  32. Content negotiation

    Content negotiation is the HTTP mechanism by which a server chooses which version of the same resource to deliver, in format, language or compression, according to the preferences the client declares in headers such as Accept or Accept-Language.

  33. Link rel=alternate header

    The Link rel=alternate header is a line of the HTTP response that announces that the same content exists in another version, at another address: in another language, if it carries hreflang, or in another format, if it carries type, such as a PDF or a Markdown version.

  34. .well-known directory

    The .well-known directory is the /.well-known/ path that RFC 8615 reserves at the root of a website for publishing information about the whole site at fixed addresses known in advance, such as /.well-known/security.txt.

  35. Core Web Vitals

    Core Web Vitals are the three metrics Google uses to measure a page's real-world experience: LCP for loading, INP for responsiveness and CLS for visual stability.

  36. What an AI agent sees on your product page

    Paste your product page source and see what a crawler that does not run JavaScript gets: text, headings, meta and JSON-LD. A free tool.

  37. robots.txt for AI bots: check and write your rules

    Check which AI bots your robots.txt lets in, GPTBot, ClaudeBot and PerplexityBot among them, and write the lines that allow or block each one.

Next, in your store

Guide

What an AI crawler sees in your store
Comparison

llms.txt, sitemap and structured data
Reading path

AI search
Reading path

Measurement and verification

Make your catalog easy for bots to read

Clione fills every product page with what bots read best: the JSON-LD graph with its structured data, the meta description and the FAQs. You review and press Push; then we check your live store so it all stays published.

See how Clione works →