# What is a web crawler and how does it work? > What a web crawler is, how it discovers your store's product pages, what it respects in robots.txt, and how to check in your logs who is visiting. URL: https://clione.ai/glossary/web-crawler/ Lang: en A web crawler has nothing to do with tracking a person, an order or a phone: it is the program that reads your site so another system can use it. Google also calls it a robot, bot or spider, and its own is called [Googlebot](/glossary/googlebot/).[^google-how-search] For a store, the consequence is easy to state: **if a crawler does not reach a [product page](/glossary/product-page/), no search engine or assistant can use it**, however good it is. And reaching it is not enough, because what the crawler reads is what the server hands it, not what you see in the editor. ## How it works Google describes its search engine in three stages: crawling, [indexing](/glossary/indexing/) and serving results. In the first, its crawlers **download the text, images and videos** of the pages they have found.[^google-how-search] Pages are discovered in two ways: because they are linked from pages already known or because the owner submits them in an [XML sitemap](/glossary/xml-sitemap/).[^google-how-search] Google defines crawling as the process of using automated software to discover new pages and understand them.[^google-about-crawling] A crawler does not request everything, or all the time. Googlebot uses an algorithmic process to decide which sites to crawl, how often and how many pages to fetch from each, and during crawling it renders pages with a recent version of Chrome.[^google-how-search] If the site slows down or returns errors, **the crawl rate drops on its own** so as not to overload the server.[^google-about-crawling] That practical limit on how much gets crawled is what is known as [crawl budget](/glossary/crawl-budget/). What the crawler does ends there. Google warns that it does not guarantee it will crawl, index or serve a page even if it meets its requirements, and that **not every page it processes ends up indexed**.[^google-how-search] Google splits its clients into three groups. Common crawlers, like Googlebot, always respect the [`robots.txt`](/glossary/robots-txt/) rules in automatic crawls. Special-case crawlers work for specific products under agreements with the site and may bypass the general rules. And user-triggered fetchers make a request when someone asks for it from a tool.[^google-crawlers] AI companies follow a similar scheme. OpenAI documents [OAI-SearchBot](/glossary/oai-searchbot/) for showing websites in [ChatGPT search](/glossary/chatgpt-search/), [GPTBot](/glossary/gptbot/) for crawling content that may be used to train its models, and [ChatGPT-User](/glossary/chatgpt-user/) for actions a person initiates, to which `robots.txt` rules may not apply.[^openai-bots] That is why **blocking one does not block the others**. The rules a crawler reads are in the Robots Exclusion Protocol, published as RFC 9309 in 2022. Three details of the standard matter in a store:[^rfc9309] - **The rules are not a form of access authorization.** An honest crawler respects them; a malicious one does not. - **A server error closes the whole door.** If `robots.txt` returns a 5xx error, the crawler has to assume it cannot request anything. If it returns a 4xx, like a 404, it may request everything. - **Changes are not immediate.** The crawler may cache the file for up to 24 hours. OpenAI also estimates about 24 hours for its systems to adjust to a change.[^openai-bots] ## In a store What a crawler leaves on your server is a line in the access log. A **fictional example**, simplified, with IPs reserved for documentation: ```text 192.0.2.10 [06/Oct/2026:09:14:02 +0000] "GET /products/travel-tripod-t20 HTTP/1.1" 200 "...Googlebot/2.1..." 198.51.100.7 [06/Oct/2026:09:20:45 +0000] "GET /robots.txt HTTP/1.1" 200 "...OAI-SearchBot..." 203.0.113.42 [06/Oct/2026:09:31:10 +0000] "GET /products/travel-tripod-t20 HTTP/1.1" 200 "...Googlebot/2.1..." ``` The three lines say less than they seem to. The name at the end is the [user agent](/glossary/user-agent/) that whoever requests the page declares, and anyone can type it. Google offers a way to check because **some visitors crawl sites pretending to be Google**.[^google-verify] Until you verify the IP, the first and third lines are two visits that claim to be Googlebot. And even if both were genuine, a 200 in the log only proves the product page was downloaded, not that it is in the index. **Blocking a page in `robots.txt` does not take it out of Google.** The file tells crawlers which URLs they may request, but it is not a mechanism for keeping a page out of Google: a blocked URL **can still be indexed if other sites link to it**, and then it appears without a description. To keep it from being indexed, Google recommends [`noindex`](/glossary/noindex/), a password or removing the page.[^google-robots-intro] ## How to check it 1. **Open `yourdomain.com/robots.txt`** and confirm it responds with a 200. A 5xx error makes a crawler that follows the standard request nothing from your site.[^rfc9309] 2. **Search your server or CDN logs** for requests from the agents you care about: Googlebot, OAI-SearchBot, GPTBot and the others each company documents. 3. **Verify they are who they say they are.** For Google: run `host` on the IP from the log, check that the name ends in `googlebot.com`, `google.com` or `googleusercontent.com`, and do the forward lookup on that name to see that it returns the same IP. You can also compare the IP with the ranges Google publishes in JSON files.[^google-verify] OpenAI publishes its agents' ranges the same way.[^openai-bots] 4. **Check whether the page is indexed** with the URL Inspection tool in [Search Console](/glossary/google-search-console/). Crawled does not mean indexed.[^google-how-search] 5. **Compare the HTML the server serves with what you see on screen.** Google renders with Chrome,[^google-how-search] but a product page's price, availability and [JSON-LD](/glossary/json-ld/) are best already in the source HTML. If they only appear after running JavaScript, read what [JavaScript rendering](/glossary/javascript-rendering/) implies. ## Related concept: indexing Crawling is reading; [indexing](/glossary/indexing/) is deciding what gets stored. Google separates it into two stages: the crawler downloads the page and then Google analyzes the text, images and videos and stores the information in its index, a huge database.[^google-how-search] A page can be crawled many times and never indexed, and one blocked in `robots.txt` can end up indexed without anyone having read it.[^google-robots-intro] The entry on [crawling vs. indexing](/glossary/crawling-vs-indexing/) goes into that difference in more detail. To keep going, the guide to [what an AI crawler sees on your store](/guides/what-an-ai-crawler-sees/) goes agent by agent through what each one reads, and the [reading path](/learn/crawling-and-indexing/) for this topic puts the rest of the terms in order.