# Crawling and indexing for your store: reading path

> Reading path on crawling and indexing for online stores: from robots.txt to AI bots and agent files, in the order they make sense.

URL: https://clione.ai/learn/crawling-and-indexing/
Lang: en

First who reads and what they ask for, then what you tell them about each URL, then who each [bot](/glossary/web-crawler/) is, and last what you publish with agents in mind. By the time you reach the AI bots, you will know what a file can and cannot ask of them. The path has five stretches.

1. **Steps 1 to 6 · How a search engine reads the web.** The [web crawler](/glossary/web-crawler/), [indexing](/glossary/indexing/) and the [difference between the two](/glossary/crawling-vs-indexing/). Then the two basic tools of the site: [robots.txt](/glossary/robots-txt/), with the [user agent](/glossary/user-agent/) each bot identifies itself with, and the [XML sitemap](/glossary/xml-sitemap/).
2. **Steps 7 to 14 · Which page counts and how many get visited.** The signals you use to choose what gets indexed and under which URL: [noindex](/glossary/noindex/), the [canonical URL](/glossary/canonical-url/), [duplicate content](/glossary/duplicate-content/), the [301 redirect](/glossary/301-redirect/), [hreflang](/glossary/hreflang/) and [internal linking](/glossary/internal-linking/). Two terms for large stores close the stretch: [faceted navigation](/glossary/faceted-navigation/), which multiplies URLs, and [crawl budget](/glossary/crawl-budget/).
3. **Steps 15 to 23 · Who crawls your store.** The search engines' crawlers, [Googlebot](/glossary/googlebot/) and [Bingbot](/glossary/bingbot/), and the AI companies' crawlers, each with its own job: [GPTBot](/glossary/gptbot/), [OAI-SearchBot](/glossary/oai-searchbot/), [ChatGPT-User](/glossary/chatgpt-user/), [PerplexityBot](/glossary/perplexitybot/) and [ClaudeBot](/glossary/claudebot/). Google's two controls for its AI close the stretch: [Google-Extended](/glossary/google-extended/) and the [generative AI control](/glossary/search-console-generative-ai-control/) in [Search Console](/glossary/google-search-console/).
4. **Steps 24 to 28 · Opening or closing the door.** [AI bot blocking](/glossary/ai-bot-blocking/) and what the network layer decides: [caching and CDNs](/glossary/cdn-caching-for-bots/), bot signing with [Web Bot Auth](/glossary/web-bot-auth/) and charging per visit with [pay per crawl](/glossary/pay-per-crawl/). [IndexNow](/glossary/indexnow/) closes it, the notice that a URL has changed.
5. **Steps 29 to 35 · For agents, and one measure apart.** The files and responses offered to AI: [llms.txt](/glossary/llms-txt/), [agents.md](/glossary/agents-md/), [Markdown for agents](/glossary/markdown-for-agents/), the [content negotiation](/glossary/content-negotiation/) that serves it, the [Link header](/glossary/link-header/) and the [`.well-known`](/glossary/well-known/) directory. The last one, [Core Web Vitals](/glossary/core-web-vitals/), is not about who gets in but about how the page loads, and it stays here because it is the one most often confused with crawling.

This is one of the [seven reading paths](/learn/). Read it after [the path on AI in search engines](/learn/ai-search/) if you do not yet know why the AI companies' bots visit you.