What is ClaudeBot, Anthropic's web crawler?

ClaudeBot is not Claude, the assistant, and it is not a feature of it: it is the crawler Anthropic uses to travel the web and gather content to improve its models. For a store, the name usually turns up in one of two ways: on a list of "AI bots" someone proposes blocking, or in the server logs, sometimes as a burst of back-to-back requests. In both cases the useful question is the same: what stops happening if you close the door on it, and what keeps happening even if you do.

How it works

Anthropic uses three agents, each with its own job, and describes them on a single help page, last updated on April 7, 2026:⁠^{[1]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?}

The practical consequence is the same as at OpenAI, where GPTBot crawls for training and OAI-SearchBot for search:⁠^{[2]OpenAI. Overview of OpenAI Crawlers} training and visibility are decided separately. Blocking ClaudeBot affects the first; for the second, the other two are what count.

Anthropic states that its bots respect "do not crawl" signals by following the standard robots.txt directives, and it gives the example of closing the whole site to ClaudeBot with a group of its own and Disallow: /. It asks you to repeat it on each subdomain you want to exclude, because each one has its own file. And it adds two things not every crawler does: it supports the Crawl-delay directive, which is not part of the standard, so you can ask it to space out its visits, and it says it will not attempt to bypass CAPTCHAs on the sites it crawls.⁠^{[1]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?}

What it advises against is blocking by IP. According to Anthropic, other methods, such as blocking IP addresses, may not work well or guarantee lasting exclusion, because they prevent it from reading your robots.txt.⁠^{[1]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?} If ClaudeBot cannot read the file, it cannot read your Disallow either.

To check whether a visit is really theirs, Anthropic publishes the IP ranges of its bots at claude.com/crawling/bots.json: if a crawler comes from an address on that list, it comes from Anthropic.⁠^{[1]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?} The version we read on October 6, 2026 was dated October 2, 2026 and had 28 IPv4 prefixes.⁠^{[3]Anthropic. bots.json (IP ranges of Anthropic's bots)} The list is one for all its bots, so it confirms the visit is from Anthropic, and the name in the user agent says which of the three it is.

Google has no direct equivalent with visits of its own. Its control for training Gemini, Google-Extended, is a name that exists only in robots.txt: the crawling is done by Google's usual agents, and Google says it does not affect a site's inclusion in Search.⁠^{[4]Google Crawling Infrastructure. Google's common crawlers} With ClaudeBot you do see requests in your logs.

In a store

A climbing gear store wants its catalog kept out of model training, but still wants to show up when someone asks Claude about a harness. This is a fictional example. Its robots.txt:

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /collections/*sort_by*

User-agent: ClaudeBot
Disallow: /

ClaudeBot finds its group and cannot request anything. Claude-SearchBot and Claude-User have no group of their own, so they follow the asterisk one: they can request product pages, but not the cart, the checkout or re-sorted collections. The standard says each crawler looks for the group that names it and, only if it finds none, obeys the asterisk group.⁠^{[5]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}

Another store does not want to shut the door, only to have ClaudeBot request fewer pages per minute. If it gives ClaudeBot a group of its own for that, it has to repeat the general rules inside it, because a crawler that finds its own group stops reading the asterisk one:⁠^{[5]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}

User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /cart
Disallow: /checkout
Disallow: /collections/*sort_by*

Without those three lines, the Crawl-delay would open the cart and the re-sorted collections to ClaudeBot.

How to check it

1. Open yourdomain.com/robots.txt and look for Anthropic's three names. If none has a group, the asterisk group applies to them.⁠^{[5]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} Do the same on every subdomain that serves pages.⁠^{[1]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?}
2. If ClaudeBot has a group of its own, check that it repeats your general rules (cart, checkout, filtered collections), because it does not inherit them.⁠^{[5]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}
3. Search for ClaudeBot, Claude-SearchBot and Claude-User in the logs of your server or CDN, and count the requests from each one separately.
4. Check each IP against claude.com/crawling/bots.json from the same day. An IP outside the list is not one Anthropic declares for its bots.⁠^{[1]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?}
5. If there is an IP block in the firewall, review it: Anthropic warns that it can keep it from reading your robots.txt and fail to guarantee the exclusion.⁠^{[1]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?}

Related concept: GPTBot

GPTBot is OpenAI's equivalent: it crawls content that may be used to train its foundation models, and blocking it signals that the site's content should not be used in that training.⁠^{[2]OpenAI. Overview of OpenAI Crawlers} Both leave search to another agent, OAI-SearchBot at OpenAI and Claude-SearchBot at Anthropic, and both have an agent beside them for a person's requests: ChatGPT-User and Claude-User.⁠^{[2]OpenAI. Overview of OpenAI Crawlers}⁠^{[1]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?} The difference is in that third agent. OpenAI warns that robots.txt rules may not apply to ChatGPT-User visits;⁠^{[2]OpenAI. Overview of OpenAI Crawlers} Anthropic says its bots respect robots.txt and that blocking Claude-User has a visibility cost.⁠^{[1]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?} If your robots.txt already handles GPTBot, handle ClaudeBot the same way and under its own name: a rule for one does not cover the other.

To go further, the guide to what an AI crawler sees in your store goes agent by agent through what each one reads, and the reading path for this topic orders the rest of the terms.