What is a web crawler?

A web crawler has nothing to do with tracking a person, an order or a phone: it is the program that reads your site so another system can use it. Google also calls it a robot, bot or spider, and its own is called Googlebot.⁠^{[1]Google Search Central. In-depth guide to how Google Search works} For a store, the consequence is easy to state: if a crawler does not reach a product page, no search engine or assistant can use it, however good it is. And reaching it is not enough, because what the crawler reads is what the server hands it, not what you see in the editor.

How it works

Google describes its search engine in three stages: crawling, indexing and serving results. In the first, its crawlers download the text, images and videos of the pages they have found.⁠^{[1]Google Search Central. In-depth guide to how Google Search works} Pages are discovered in two ways: because they are linked from pages already known or because the owner submits them in an XML sitemap.⁠^{[1]Google Search Central. In-depth guide to how Google Search works} Google defines crawling as the process of using automated software to discover new pages and understand them.⁠^{[2]Google Crawling Infrastructure. Things to know about Google's web crawling}

A crawler does not request everything, or all the time. Googlebot uses an algorithmic process to decide which sites to crawl, how often and how many pages to fetch from each, and during crawling it renders pages with a recent version of Chrome.⁠^{[1]Google Search Central. In-depth guide to how Google Search works} If the site slows down or returns errors, the crawl rate drops on its own so as not to overload the server.⁠^{[2]Google Crawling Infrastructure. Things to know about Google's web crawling} That practical limit on how much gets crawled is what is known as crawl budget.

What the crawler does ends there. Google warns that it does not guarantee it will crawl, index or serve a page even if it meets its requirements, and that not every page it processes ends up indexed.⁠^{[1]Google Search Central. In-depth guide to how Google Search works}

Google splits its clients into three groups. Common crawlers, like Googlebot, always respect the robots.txt rules in automatic crawls. Special-case crawlers work for specific products under agreements with the site and may bypass the general rules. And user-triggered fetchers make a request when someone asks for it from a tool.⁠^{[3]Google Crawling Infrastructure. Google crawler (user agent) overview} AI companies follow a similar scheme. OpenAI documents OAI-SearchBot for showing websites in ChatGPT search, GPTBot for crawling content that may be used to train its models, and ChatGPT-User for actions a person initiates, to which robots.txt rules may not apply.⁠^{[4]OpenAI. Overview of OpenAI Crawlers} That is why blocking one does not block the others.

The rules a crawler reads are in the Robots Exclusion Protocol, published as RFC 9309 in 2022. Three details of the standard matter in a store:⁠^{[5]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}

• The rules are not a form of access authorization. An honest crawler respects them; a malicious one does not.
• A server error closes the whole door. If robots.txt returns a 5xx error, the crawler has to assume it cannot request anything. If it returns a 4xx, like a 404, it may request everything.
• Changes are not immediate. The crawler may cache the file for up to 24 hours. OpenAI also estimates about 24 hours for its systems to adjust to a change.⁠^{[4]OpenAI. Overview of OpenAI Crawlers}

In a store

What a crawler leaves on your server is a line in the access log. A fictional example, simplified, with IPs reserved for documentation:

192.0.2.10   [06/Oct/2026:09:14:02 +0000] "GET /products/travel-tripod-t20 HTTP/1.1" 200  "...Googlebot/2.1..."
198.51.100.7 [06/Oct/2026:09:20:45 +0000] "GET /robots.txt HTTP/1.1"                 200  "...OAI-SearchBot..."
203.0.113.42 [06/Oct/2026:09:31:10 +0000] "GET /products/travel-tripod-t20 HTTP/1.1" 200  "...Googlebot/2.1..."

The three lines say less than they seem to. The name at the end is the user agent that whoever requests the page declares, and anyone can type it. Google offers a way to check because some visitors crawl sites pretending to be Google.⁠^{[6]Google Search Central. Verifying Googlebot and other Google crawlers} Until you verify the IP, the first and third lines are two visits that claim to be Googlebot. And even if both were genuine, a 200 in the log only proves the product page was downloaded, not that it is in the index.

How to check it

1. Open yourdomain.com/robots.txt and confirm it responds with a 200. A 5xx error makes a crawler that follows the standard request nothing from your site.⁠^{[5]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}
2. Search your server or CDN logs for requests from the agents you care about: Googlebot, OAI-SearchBot, GPTBot and the others each company documents.
3. Verify they are who they say they are. For Google: run host on the IP from the log, check that the name ends in googlebot.com, google.com or googleusercontent.com, and do the forward lookup on that name to see that it returns the same IP. You can also compare the IP with the ranges Google publishes in JSON files.⁠^{[6]Google Search Central. Verifying Googlebot and other Google crawlers} OpenAI publishes its agents' ranges the same way.⁠^{[4]OpenAI. Overview of OpenAI Crawlers}
4. Check whether the page is indexed with the URL Inspection tool in Search Console. Crawled does not mean indexed.⁠^{[1]Google Search Central. In-depth guide to how Google Search works}
5. Compare the HTML the server serves with what you see on screen. Google renders with Chrome,⁠^{[1]Google Search Central. In-depth guide to how Google Search works} but a product page's price, availability and JSON-LD are best already in the source HTML. If they only appear after running JavaScript, read what JavaScript rendering implies.

Related concept: indexing

Crawling is reading; indexing is deciding what gets stored. Google separates it into two stages: the crawler downloads the page and then Google analyzes the text, images and videos and stores the information in its index, a huge database.⁠^{[1]Google Search Central. In-depth guide to how Google Search works} A page can be crawled many times and never indexed, and one blocked in robots.txt can end up indexed without anyone having read it.⁠^{[7]Google Search Central. Introduction to robots.txt} The entry on crawling vs. indexing goes into that difference in more detail.

To keep going, the guide to what an AI crawler sees on your store goes agent by agent through what each one reads, and the reading path for this topic puts the rest of the terms in order.