What is PerplexityBot, Perplexity's crawler?

PerplexityBot is the name that the crawler of Perplexity, the AI search engine of the same name, shows up as on your server. It is not the assistant or a feature you install: it is the program that walks websites so Perplexity can show them and link to them. For a store it matters for a concrete reason: it is the agent that decides whether you appear in Perplexity's results, and it is easy to block it by accident, tucked into a list of "AI bots" meant to stop model training.

How it works

Perplexity describes its agents on a page of its documentation. It says two things about PerplexityBot: that it is designed to surface and link websites in Perplexity's search results, and that it is not used to crawl content for AI foundation models. To make sure your site appears in those results, it recommends allowing it in your robots.txt and accepting requests that arrive from its published IPs.⁠^{[1]Perplexity. Perplexity Crawlers}

Each agent is managed separately in robots.txt, and Perplexity warns that a change can take up to 24 hours to be reflected in its systems.⁠^{[1]Perplexity. Perplexity Crawlers} It is a similar window to the one the standard gives for caching the file: a crawler should not use the stored copy for more than 24 hours.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}

The request arrives with this user agent, in the version Perplexity documents in October 2026:⁠^{[1]Perplexity. Perplexity Crawlers}

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

In robots.txt the short name, PerplexityBot, is enough. The standard says each crawler looks for the group that names it, ignoring case, and only if it finds none does it follow the User-agent: * group.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}

Alongside PerplexityBot works a second agent, Perplexity-User, which serves user actions: when someone asks a question, it may visit a page to give an exact answer and link it. Perplexity says it does not crawl the web or collect content to train models, and that, because a user requests the visit, it generally ignores robots.txt rules.⁠^{[1]Perplexity. Perplexity Crawlers} In practice, PerplexityBot decides whether your store is among what Perplexity can show, and Perplexity-User is the one that opens a specific product page at the moment of a question.

To tell a real visit from one that just uses the name, Perplexity publishes each agent's addresses in a JSON file, and asks you to always use the current version, because they are updated regularly.⁠^{[1]Perplexity. Perplexity Crawlers} The PerplexityBot one we read on October 6, 2026 was dated February 7, 2025 and had 8 standalone IPv4 addresses, each a /32 prefix.⁠^{[3]Perplexity. perplexitybot.json (PerplexityBot IP addresses)} The Perplexity-User one was dated October 17, 2025 and had 4.⁠^{[4]Perplexity. perplexity-user.json (Perplexity-User IP addresses)}

Perplexity also gives instructions for web application firewalls, and the idea holds for any of them: create a rule that combines the user agent name and the source IP, with the IPs taken from its official files, and give it priority over the rules that block. It spells it out for Cloudflare WAF and for AWS WAF.⁠^{[1]Perplexity. Perplexity Crawlers} The name alone can be faked; the name and the IP together, much less so.

In a store

A natural cosmetics store copies a list from a forum to "protect itself from AI" and pastes it into its robots.txt. It is a made-up example:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /

The intent was not to hand over its catalog to train models. With GPTBot and ClaudeBot the rule does that: OpenAI uses the first to gather content to train its models, and Anthropic the second to collect content to improve theirs.⁠^{[5]OpenAI. Overview of OpenAI Crawlers}⁠^{[6]Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?} With PerplexityBot it does not: Perplexity says it does not use it to train, so the only consequence of blocking it is dropping out of its search results.⁠^{[1]Perplexity. Perplexity Crawlers} And the questions a customer asks about that store can still send Perplexity-User to its product pages, because that agent generally does not follow robots.txt.⁠^{[1]Perplexity. Perplexity Crawlers}

The version that does what the store wanted takes PerplexityBot out of the group:

User-agent: *
Disallow: /cart
Disallow: /checkout

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

PerplexityBot has no group of its own, so it follows the asterisk one: it can request the product pages, but not the cart or the checkout.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol} If the store uses a firewall with rules against bots, one step is missing: allowing PerplexityBot by name and IP, as Perplexity recommends.⁠^{[1]Perplexity. Perplexity Crawlers}

How to check it

1. Open yourdomain.com/robots.txt and look for a group that names PerplexityBot. If there is none, the asterisk group applies: see what it says about the path of your product pages.⁠^{[2]IETF (Koster, Illyes, Zeller and Sassman). RFC 9309: Robots Exclusion Protocol}
2. Review the firewall or CDN. If you filter bots, check that a rule lets PerplexityBot through by name and by IP, with priority over the blocking ones.⁠^{[1]Perplexity. Perplexity Crawlers}
3. Look for PerplexityBot and Perplexity-User in the logs of your server or CDN, and count them separately.
4. Check each IP against its file: perplexitybot.json for the crawler and perplexity-user.json for user visits, in the current version.⁠^{[3]Perplexity. perplexitybot.json (PerplexityBot IP addresses)}⁠^{[4]Perplexity. perplexity-user.json (Perplexity-User IP addresses)}
5. If you changed something, wait at least a day before drawing conclusions: Perplexity allows up to 24 hours for a robots.txt change to show.⁠^{[1]Perplexity. Perplexity Crawlers}

Related concept: OAI-SearchBot

PerplexityBot is more like OAI-SearchBot than like GPTBot. Both are search agents: OpenAI uses OAI-SearchBot to surface sites in ChatGPT search results, and sites that block it do not appear in the answers of that search.⁠^{[5]OpenAI. Overview of OpenAI Crawlers} And both have next to them an agent for a person's requests, ChatGPT-User at OpenAI and Perplexity-User at Perplexity, to which robots.txt may not apply.⁠^{[5]OpenAI. Overview of OpenAI Crawlers}⁠^{[1]Perplexity. Perplexity Crawlers} The difference is training. OpenAI has a separate crawler for that, GPTBot;⁠^{[5]OpenAI. Overview of OpenAI Crawlers} Perplexity documents none and says its two agents do not collect content to train foundation models.⁠^{[1]Perplexity. Perplexity Crawlers} That is why, in a blocklist meant against training, PerplexityBot has no place.

To go further, the guide on what an AI crawler sees in your store goes agent by agent through what each one reads, and the reading path for this topic orders the rest of the terms.