# What Is Cloudflare Pay Per Crawl and How Does It Work? > What Cloudflare's pay per crawl is: how it charges AI crawlers with the 402 error, who pays, what is still in beta and what it means for a store. URL: https://clione.ai/glossary/pay-per-crawl/ Lang: en _Pay per crawl_ has nothing to do with pay-per-click advertising: here the one who pays is an AI company's [crawler](/glossary/web-crawler/), and the one who gets paid is the publisher. It is a Cloudflare feature, not a web standard, and it is only available to domains that go through its network. It matters to a store for a concrete reason: **it gives you a third option between letting AI bots in for free and shutting the door on them**, and that option has different consequences depending on which bot gets the bill. ## How it works Cloudflare introduced _pay per crawl_ on July 1, 2025, in private beta.[^cf-ppc-blog] The same day it announced that it would block by default AI crawlers that access content without permission or compensation, and that AI companies could declare whether their crawlers are for training, inference or search.[^cf-press] **In the documentation updated on July 28, 2026, the feature is still in closed beta**: to get in you fill out a form or, if you are an Enterprise customer, talk to your account manager.[^cf-ppc-what] The owner picks an action for each crawler Cloudflare identifies: **allow** (free access), **charge** (the configured price for each successful access) or **block** (no access).[^cf-ppc-select] The price is a single one for all the crawlers being charged: you can have different actions per bot, but not different prices per bot.[^cf-ppc-faq] The minimum price is $0.001 per page.[^cf-ppc-price] Since June 16, 2026 it can vary by page: the origin server, or a Cloudflare Worker, adds a `crawler-price` header to the response with that URL's price.[^cf-aicc-changelog][^cf-ppc-advanced] The conversation between the crawler and the site uses an HTTP code that had gone unused for years, **402 Payment Required**.[^cf-ppc-blog] There are two paths:[^cf-ppc-crawl] - **Ask first.** The crawler requests the page without offering anything, gets a 402 with the `crawler-price` header and repeats the request with `crawler-exact-price` and that same amount. - **Offer first.** The crawler sends `crawler-max-price` from the start with the most it is willing to pay. If the page's price is equal or lower, it gets the content; if it is higher, it gets the 402. When the page is served, the 200 response carries the `crawler-charged` header with what it will be charged.[^cf-ppc-crawl] Every visit counts, including the second time it requests the same page; error responses are not charged; and some paths are always free, among them [`/robots.txt`](/glossary/robots-txt/) and [`/sitemap.xml`](/glossary/xml-sitemap/).[^cf-ppc-faq] **The name is not enough to pay.** So that nobody can pass as a paying crawler, Cloudflare requires the crawler to sign its requests with [Web Bot Auth](/glossary/web-bot-auth/): it generates a key pair, publishes the public one and registers with its [user agent](/glossary/user-agent/).[^cf-ppc-blog] Since December 10, 2025, the payment header has to be inside the signature, so nobody tampers with it on the way.[^cf-aicc-changelog] Cloudflare charges the crawler's operator, acts as the seller (_merchant of record_) and pays sites every month through a Stripe Connect account created from its dashboard.[^cf-ppc-what][^cf-ppc-payouts] Two limits it acknowledges: the accumulated balance does not yet show in the dashboard, and payouts have settlement periods and a minimum.[^cf-ppc-payouts] ## In a store This is how the exchange would look for a charged [product page](/glossary/product-page/). It is a **made-up example**, with the headers Cloudflare documents and the Web Bot Auth signature left out: ```text GET /products/travel-tripod-t20 HTTP/2 → HTTP/2 402 crawler-price: USD 0.01 GET /products/travel-tripod-t20 HTTP/2 crawler-exact-price: USD 0.01 → HTTP/2 200 crawler-charged: USD 0.01 ``` For a news outlet, that bill is the business model. For a store, the math changes, because **what it wants is for assistants to read its product pages and recommend them**. Cloudflare warns that charging or blocking crawlers in the search engine category can hurt the site's SEO, because it can prevent good [indexing](/glossary/indexing/) of the content.[^cf-ppc-select] And on the assistants' side something similar happens: OpenAI says sites that exclude [OAI-SearchBot](/glossary/oai-searchbot/) are not shown in [ChatGPT search](/glossary/chatgpt-search/) answers, although they can still appear as navigational links.[^openai-bots] That is why the decision is made bot by bot. Training bots, such as [GPTBot](/glossary/gptbot/), are the natural candidates to pay or stay out; search bots are the ones that bring citations. Cloudflare also suggests leaving free the pages that help discover the rest, such as the home page or the categories, and starting with a small section of the site.[^cf-ppc-advanced] Believing that charging is a soft way of letting bots in. Cloudflare lets you set "charge" even for a crawler that has no payment agreement with them and therefore cannot pay: **that crawler gets the 402 and goes without the content, just as with a block**, with the difference that the message tells it there could be an agreement in the future.[^cf-ppc-blog] And if a firewall (WAF) or bot management rule blocks a crawler, that rule takes precedence over the charge.[^cf-ppc-what] ## How to check it 1. **Confirm that your domain goes through Cloudflare.** It is a requirement: the domain has to use its nameservers or have its DNS records managed by Cloudflare, and an account administrator is needed to turn the feature on.[^cf-ppc-what] 2. **Open AI Crawl Control and review each crawler's action.** The category column flags which ones are search engines, the ones Cloudflare warns not to charge or block lightly.[^cf-ppc-select] 3. **Look for WAF or bot management rules** that affect those same crawlers. If they block, the charge never gets applied.[^cf-ppc-what] 4. **Open your `robots.txt`** and check that it says the same as the dashboard. It is a request that honest bots respect; Cloudflare's dashboard acts at the network. 5. **Read the changelog before deciding.** It is a beta and it changes: the discovery API, free paths by pattern and per-page prices came after the launch.[^cf-aicc-changelog] ## Related concept: AI bot blocking [AI bot blocking](/glossary/ai-bot-blocking/) is deciding which agents cannot read your site, whether with a rule in `robots.txt` or a rule on the network. _Pay per crawl_ starts from that same list and adds a price: the crawler that pays gets in, the one that does not pay stays out. **Blocking says "no"; charging says "no, unless you pay"**, and for the bot that cannot pay the two answers are the same.[^cf-ppc-blog] Which of the two suits each bot depends on whether that bot brings you citations or just takes content. To go further, the [crawling and indexing reading path](/learn/crawling-and-indexing/) orders the rest of the terms, and the [user agent](/glossary/user-agent/) entry explains the name each crawler introduces itself with.