What is pay per crawl from Cloudflare?

Pay per crawl has nothing to do with pay-per-click advertising: here the one who pays is an AI company's crawler, and the one who gets paid is the publisher. It is a Cloudflare feature, not a web standard, and it is only available to domains that go through its network. It matters to a store for a concrete reason: it gives you a third option between letting AI bots in for free and shutting the door on them, and that option has different consequences depending on which bot gets the bill.

How it works

Cloudflare introduced pay per crawl on July 1, 2025, in private beta.⁠^{[1]Cloudflare, blog (Will Allen and Simon Newton). Introducing pay per crawl: Enabling content owners to charge AI crawlers for access} The same day it announced that it would block by default AI crawlers that access content without permission or compensation, and that AI companies could declare whether their crawlers are for training, inference or search.⁠^{[2]Cloudflare, press release. Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large} In the documentation updated on July 28, 2026, the feature is still in closed beta: to get in you fill out a form or, if you are an Enterprise customer, talk to your account manager.⁠^{[3]Cloudflare Docs, AI Crawl Control. What is Pay Per Crawl?}

The owner picks an action for each crawler Cloudflare identifies: allow (free access), charge (the configured price for each successful access) or block (no access).⁠^{[4]Cloudflare Docs, AI Crawl Control. Select crawlers to charge} The price is a single one for all the crawlers being charged: you can have different actions per bot, but not different prices per bot.⁠^{[5]Cloudflare Docs, AI Crawl Control. Pay Per Crawl FAQ} The minimum price is $0.001 per page.⁠^{[6]Cloudflare Docs, AI Crawl Control. Set a pay per crawl price} Since June 16, 2026 it can vary by page: the origin server, or a Cloudflare Worker, adds a crawler-price header to the response with that URL's price.⁠^{[7]Cloudflare Docs. AI Crawl Control changelog}⁠^{[8]Cloudflare Docs, AI Crawl Control. Advanced configuration}

The conversation between the crawler and the site uses an HTTP code that had gone unused for years, 402 Payment Required.⁠^{[1]Cloudflare, blog (Will Allen and Simon Newton). Introducing pay per crawl: Enabling content owners to charge AI crawlers for access} There are two paths:⁠^{[9]Cloudflare Docs, AI Crawl Control. Crawl pages}

• Ask first. The crawler requests the page without offering anything, gets a 402 with the crawler-price header and repeats the request with crawler-exact-price and that same amount.
• Offer first. The crawler sends crawler-max-price from the start with the most it is willing to pay. If the page's price is equal or lower, it gets the content; if it is higher, it gets the 402.

When the page is served, the 200 response carries the crawler-charged header with what it will be charged.⁠^{[9]Cloudflare Docs, AI Crawl Control. Crawl pages} Every visit counts, including the second time it requests the same page; error responses are not charged; and some paths are always free, among them /robots.txt and /sitemap.xml.⁠^{[5]Cloudflare Docs, AI Crawl Control. Pay Per Crawl FAQ}

The name is not enough to pay. So that nobody can pass as a paying crawler, Cloudflare requires the crawler to sign its requests with Web Bot Auth: it generates a key pair, publishes the public one and registers with its user agent.⁠^{[1]Cloudflare, blog (Will Allen and Simon Newton). Introducing pay per crawl: Enabling content owners to charge AI crawlers for access} Since December 10, 2025, the payment header has to be inside the signature, so nobody tampers with it on the way.⁠^{[7]Cloudflare Docs. AI Crawl Control changelog}

Cloudflare charges the crawler's operator, acts as the seller (merchant of record) and pays sites every month through a Stripe Connect account created from its dashboard.⁠^{[3]Cloudflare Docs, AI Crawl Control. What is Pay Per Crawl?}⁠^{[10]Cloudflare Docs, AI Crawl Control. Manage payouts} Two limits it acknowledges: the accumulated balance does not yet show in the dashboard, and payouts have settlement periods and a minimum.⁠^{[10]Cloudflare Docs, AI Crawl Control. Manage payouts}

In a store

This is how the exchange would look for a charged product page. It is a made-up example, with the headers Cloudflare documents and the Web Bot Auth signature left out:

GET /products/travel-tripod-t20 HTTP/2
→ HTTP/2 402
  crawler-price: USD 0.01

GET /products/travel-tripod-t20 HTTP/2
  crawler-exact-price: USD 0.01
→ HTTP/2 200
  crawler-charged: USD 0.01

For a news outlet, that bill is the business model. For a store, the math changes, because what it wants is for assistants to read its product pages and recommend them. Cloudflare warns that charging or blocking crawlers in the search engine category can hurt the site's SEO, because it can prevent good indexing of the content.⁠^{[4]Cloudflare Docs, AI Crawl Control. Select crawlers to charge} And on the assistants' side something similar happens: OpenAI says sites that exclude OAI-SearchBot are not shown in ChatGPT search answers, although they can still appear as navigational links.⁠^{[11]OpenAI. Overview of OpenAI Crawlers}

That is why the decision is made bot by bot. Training bots, such as GPTBot, are the natural candidates to pay or stay out; search bots are the ones that bring citations. Cloudflare also suggests leaving free the pages that help discover the rest, such as the home page or the categories, and starting with a small section of the site.⁠^{[8]Cloudflare Docs, AI Crawl Control. Advanced configuration}

How to check it

1. Confirm that your domain goes through Cloudflare. It is a requirement: the domain has to use its nameservers or have its DNS records managed by Cloudflare, and an account administrator is needed to turn the feature on.⁠^{[3]Cloudflare Docs, AI Crawl Control. What is Pay Per Crawl?}
2. Open AI Crawl Control and review each crawler's action. The category column flags which ones are search engines, the ones Cloudflare warns not to charge or block lightly.⁠^{[4]Cloudflare Docs, AI Crawl Control. Select crawlers to charge}
3. Look for WAF or bot management rules that affect those same crawlers. If they block, the charge never gets applied.⁠^{[3]Cloudflare Docs, AI Crawl Control. What is Pay Per Crawl?}
4. Open your robots.txt and check that it says the same as the dashboard. It is a request that honest bots respect; Cloudflare's dashboard acts at the network.
5. Read the changelog before deciding. It is a beta and it changes: the discovery API, free paths by pattern and per-page prices came after the launch.⁠^{[7]Cloudflare Docs. AI Crawl Control changelog}

Related concept: AI bot blocking

AI bot blocking is deciding which agents cannot read your site, whether with a rule in robots.txt or a rule on the network. Pay per crawl starts from that same list and adds a price: the crawler that pays gets in, the one that does not pay stays out. Blocking says "no"; charging says "no, unless you pay", and for the bot that cannot pay the two answers are the same.⁠^{[1]Cloudflare, blog (Will Allen and Simon Newton). Introducing pay per crawl: Enabling content owners to charge AI crawlers for access} Which of the two suits each bot depends on whether that bot brings you citations or just takes content.

To go further, the crawling and indexing reading path orders the rest of the terms, and the user agent entry explains the name each crawler introduces itself with.