# What Is ClaudeBot, Anthropic's Web Crawler?

> What ClaudeBot is, the crawler Anthropic uses to collect content for training Claude, what changes if you block it and how to verify its visits.

URL: https://clione.ai/glossary/claudebot/
Lang: en

ClaudeBot is not Claude, the assistant, and it is not a feature of it: it is the [crawler](/glossary/web-crawler/) Anthropic uses to travel the web and gather content to improve its models. For a store, the name usually turns up in one of two ways: on a list of "AI bots" someone proposes blocking, or in the server logs, sometimes as a burst of back-to-back requests. In both cases the useful question is the same: **what stops happening if you close the door on it, and what keeps happening even if you do**.

## How it works

Anthropic uses three agents, each with its own job, and describes them on a single help page, last updated on April 7, 2026:[^anthropic-bots]

| Agent            | What Anthropic uses it for                                                                       | What happens if you block it                                                                                          |
| ---------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- |
| ClaudeBot        | Collecting content from the web to improve the usefulness and safety of its generative AI models | The site signals that its future content should be excluded from training data                                        |
| Claude-SearchBot | Crawling the web to improve the quality of search results                                        | Claude does not index your content for search, which **may reduce your visibility** and the accuracy of what it shows |
| Claude-User      | Visiting a website when a person asks a question in Claude                                       | Claude cannot retrieve your content to answer, which may reduce your visibility in the searches users request         |

The practical consequence is the same as at OpenAI, where [GPTBot](/glossary/gptbot/) crawls for training and [OAI-SearchBot](/glossary/oai-searchbot/) for search:[^openai-bots] **training and visibility are decided separately**. Blocking ClaudeBot affects the first; for the second, the other two are what count.

Anthropic states that its bots respect "do not crawl" signals by following the standard [robots.txt](/glossary/robots-txt/) directives, and it gives the example of closing the whole site to ClaudeBot with a group of its own and `Disallow: /`. It asks you to repeat it **on each subdomain** you want to exclude, because each one has its own file. And it adds two things not every crawler does: it supports the `Crawl-delay` directive, which is not part of the standard, so you can ask it to space out its visits, and it says it **will not attempt to bypass CAPTCHAs** on the sites it crawls.[^anthropic-bots]

What it advises against is blocking by IP. According to Anthropic, other methods, such as blocking IP addresses, may not work well or guarantee lasting exclusion, **because they prevent it from reading your robots.txt**.[^anthropic-bots] If ClaudeBot cannot read the file, it cannot read your `Disallow` either.

To check whether a visit is really theirs, Anthropic publishes the IP ranges of its bots at `claude.com/crawling/bots.json`: if a crawler comes from an address on that list, it comes from Anthropic.[^anthropic-bots] The version we read on October 6, 2026 was dated October 2, 2026 and had 28 IPv4 prefixes.[^anthropic-ips] **The list is one for all its bots**, so it confirms the visit is from Anthropic, and the name in the [user agent](/glossary/user-agent/) says which of the three it is.

Google has no direct equivalent with visits of its own. Its control for training [Gemini](/glossary/gemini/), [Google-Extended](/glossary/google-extended/), is a name that exists only in robots.txt: the crawling is done by Google's usual agents, and Google says it does not affect a site's inclusion in Search.[^google-common-crawlers] With ClaudeBot you do see requests in your logs.

## In a store

A climbing gear store wants its catalog kept out of model training, but still wants to show up when someone asks Claude about a harness. This is a **fictional example**. Its robots.txt:

```text
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /collections/*sort_by*

User-agent: ClaudeBot
Disallow: /
```

ClaudeBot finds its group and cannot request anything. Claude-SearchBot and Claude-User have no group of their own, so they follow the asterisk one: they can request product pages, but not the cart, the checkout or re-sorted collections. The standard says each crawler looks for the group that names it and, only if it finds none, obeys the asterisk group.[^rfc9309]

Another store does not want to shut the door, only to have ClaudeBot request fewer pages per minute. If it gives ClaudeBot a group of its own for that, it has to repeat the general rules inside it, because **a crawler that finds its own group stops reading the asterisk one**:[^rfc9309]

```text
User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /cart
Disallow: /checkout
Disallow: /collections/*sort_by*
```

Without those three lines, the `Crawl-delay` would open the cart and the re-sorted collections to ClaudeBot.


**Blocking ClaudeBot does not take you out of Claude, and blocking "everything from Anthropic" can.** ClaudeBot is the training agent. If Claude-SearchBot and Claude-User end up on the same list, Anthropic warns that this **may reduce your visibility** in Claude's search results and in answers to its users' questions.[^anthropic-bots] Before you copy a list of AI bots, look at which names it carries.


## How to check it

1. **Open `yourdomain.com/robots.txt` and look for Anthropic's three names.** If none has a group, the asterisk group applies to them.[^rfc9309] Do the same on every subdomain that serves pages.[^anthropic-bots]
2. **If ClaudeBot has a group of its own, check that it repeats your general rules** (cart, checkout, filtered collections), because it does not inherit them.[^rfc9309]
3. **Search for ClaudeBot, Claude-SearchBot and Claude-User in the logs** of your server or CDN, and count the requests from each one separately.
4. **Check each IP against `claude.com/crawling/bots.json`** from the same day. An IP outside the list is not one Anthropic declares for its bots.[^anthropic-bots]
5. **If there is an IP block in the firewall, review it**: Anthropic warns that it can keep it from reading your robots.txt and fail to guarantee the exclusion.[^anthropic-bots]

## Related concept: GPTBot

GPTBot is OpenAI's equivalent: it crawls content that may be used to train its foundation models, and blocking it signals that the site's content should not be used in that training.[^openai-bots] Both leave search to another agent, OAI-SearchBot at OpenAI and Claude-SearchBot at Anthropic, and both have an agent beside them for a person's requests: [ChatGPT-User](/glossary/chatgpt-user/) and Claude-User.[^openai-bots][^anthropic-bots] The difference is in that third agent. OpenAI warns that **robots.txt rules may not apply** to ChatGPT-User visits;[^openai-bots] Anthropic says its bots respect robots.txt and that blocking Claude-User has a visibility cost.[^anthropic-bots] If your robots.txt already handles GPTBot, handle ClaudeBot the same way and under its own name: **a rule for one does not cover the other**.

To go further, the guide to [what an AI crawler sees in your store](/guides/what-an-ai-crawler-sees/) goes agent by agent through what each one reads, and the [reading path](/learn/crawling-and-indexing/) for this topic orders the rest of the terms.