What is Web Bot Auth and how does a bot sign its requests?

Today a bot introduces itself with its user agent, a name it writes itself. That is why Google explains how to verify its crawlers: for the case where someone accesses your site pretending to be Google.⁠^{[1]Google Search Central. Verifying Googlebot and other Google crawlers} Web Bot Auth is the proposal to close that gap with cryptography: the bot signs every request and your server, or your CDN, checks the signature. It matters to a store for a practical reason: the decision about who to let in is increasingly made by a firewall, and an agent that shops on behalf of a customer and a scraper disguised as one should not get the same treatment.

How it works

The base is an already published standard, RFC 9421 (February 2024), which describes how to create, encode and verify digital signatures over parts of an HTTP message. It defines two headers: Signature-Input, which says which parts of the message the signature covers and with what parameters, and Signature, which carries the signature. Its goal is to provide end-to-end integrity and authenticity for those parts, even when the message passes through intermediaries.⁠^{[2]IETF, RFC Editor (February 2024). RFC 9421: HTTP Message Signatures}

Web Bot Auth is a profile of that standard for automated traffic. The working group's document, HTTP Message Signatures for automated traffic, by Thibault Meunier (Cloudflare) and Sandor Major (Google), describes a protocol for automated HTTP clients to identify themselves cryptographically, and adds three pieces:⁠^{[3]IETF, webbotauth working group (Meunier and Major, September 1, 2026). HTTP Message Signatures for automated traffic (draft-ietf-webbotauth-httpsig-protocol-00)}

• The Signature-Agent header, which points to an HTTPS URL where the signer's keys are.
• A key directory in JWKS format (JSON Web Key Set), published at /.well-known/http-message-signatures-directory on the signer's domain, served over HTTPS and only with a 200. It is a path in the .well-known directory.
• Rules for the signature: the tag tag="web-bot-auth", the signature has to cover at least the destination domain (@authority or @target-uri), and it carries a creation time, an expiry time (a maximum of 24 hours is recommended) and the key identifier. An optional nonce protects against reuse of a signed request.

To verify, whoever receives the request reads Signature-Agent, downloads the key directory, looks up the key by its identifier and checks the signature according to RFC 9421.⁠^{[3]IETF, webbotauth working group (Meunier and Major, September 1, 2026). HTTP Message Signatures for automated traffic (draft-ietf-webbotauth-httpsig-protocol-00)}

Where it stands

The IETF created a working group, webbotauth, to standardize methods that cryptographically authenticate automated clients and give more information about who operates them. Its charter leaves out, among other things, authenticating an agent's end user, tracking bots or assigning them reputation, and telling bots that do not participate apart from people. The schedule called for the authentication specifications by April 30, 2026.⁠^{[4]IETF Datatracker. Web Bot Auth (webbotauth): working group charter}

The working group document came later: it is version 00, from September 1, 2026, and expires on March 5, 2027. The two individual drafts that preceded it, the architecture one and the directory one, are listed as replaced, with their last version from March 2, 2026.⁠^{[5]IETF Datatracker. Web Bot Auth (webbotauth): working group documents}⁠^{[3]IETF, webbotauth working group (Meunier and Major, September 1, 2026). HTTP Message Signatures for automated traffic (draft-ietf-webbotauth-httpsig-protocol-00)} And it is an Internet-Draft: a working document valid for six months at most, which can be updated, replaced or made obsolete at any time.⁠^{[3]IETF, webbotauth working group (Meunier and Major, September 1, 2026). HTTP Message Signatures for automated traffic (draft-ietf-webbotauth-httpsig-protocol-00)}

Who already uses it

• Google. It signs with Web Bot Auth part of the requests from Google-Agent, the agent that systems hosted on Google's infrastructure use to browse the web and take actions at a user's request. It marks it as experimental, because the specification is a draft that may change, and recommends that you keep checking the IP, the reverse DNS and the user agent too while it extends signed traffic.⁠^{[6]Google Crawling Infrastructure. Authenticate requests with Web Bot Auth (experimental)} As a user-triggered fetcher, Google-Agent generally does not follow robots.txt rules.⁠^{[7]Google Crawling Infrastructure. Google's user-triggered fetchers}
• Cloudflare. It uses Web Bot Auth as one of the methods to verify signed bots and agents, requires Ed25519 keys and lets the bot's operator register its key directory from the dashboard. According to its documentation, the site owner does not have to do anything: Cloudflare validates the signatures of registered bots.⁠^{[8]Cloudflare, Bots documentation. Web Bot Auth} Its documentation classifies verified bots by how they are operated: directly or through an intermediary.⁠^{[9]Cloudflare, Bots documentation. Signed agents}

In a store

This is how a signed request from Google-Agent would reach the product page of a fictional example store. The structure follows the draft's example and the value of Signature-Agent is the one Google documents; the dates, the key identifier and the signature are made up:⁠^{[3]IETF, webbotauth working group (Meunier and Major, September 1, 2026). HTTP Message Signatures for automated traffic (draft-ietf-webbotauth-httpsig-protocol-00)}⁠^{[6]Google Crawling Infrastructure. Authenticate requests with Web Bot Auth (experimental)}

GET /products/moka-steel-6 HTTP/1.1
Host: coffee-makers.example
User-Agent: Mozilla/5.0 (X11; Linux x86_64) ... (compatible; Google-Agent)
Signature-Agent: g="https://agent.bot.goog"
Signature-Input: g=("@authority" "signature-agent";key="g");created=1791273600;expires=1791277200;keyid="example-key";tag="web-bot-auth"
Signature: g=:ZXhhbXBsZS1zaWduYXR1cmU=:

Line by line: User-Agent is what the bot says it is and proves nothing. Signature-Agent says where its keys are: https://agent.bot.goog/.well-known/http-message-signatures-directory.⁠^{[6]Google Crawling Infrastructure. Authenticate requests with Web Bot Auth (experimental)} Signature-Input says the signature covers the store's domain and the Signature-Agent header itself, valid for one hour. If someone copies the request to another site, the signature stops being valid, because it covered coffee-makers.example. If someone makes up the signature, they cannot generate it without the private key.

How to check it

1. Ask your CDN or your platform whether it verifies Web Bot Auth. If you are behind Cloudflare, Cloudflare does the verification with registered bots.⁠^{[8]Cloudflare, Bots documentation. Web Bot Auth}
2. Look for the Signature-Agent header in your server or CDN logs, if they keep request headers. Its value tells you which signer the request declares.
3. Open a key directory, for example Google's at https://agent.bot.goog/.well-known/http-message-signatures-directory, to see what a signer publishes.⁠^{[6]Google Crawling Infrastructure. Authenticate requests with Web Bot Auth (experimental)}
4. Keep verifying with reverse DNS and IP ranges for Googlebot and Google's other crawlers.⁠^{[1]Google Search Central. Verifying Googlebot and other Google crawlers}
5. Check the draft's status on the IETF Datatracker before building anything custom: the current version is 00, from September 2026.⁠^{[5]IETF Datatracker. Web Bot Auth (webbotauth): working group documents}

Related concept: user agent

The user agent is a statement; Web Bot Auth is proof. Both travel in the same request and answer the same question, who is calling, with different degrees of trust: the first is written by the client, the second can only be produced by whoever holds the key.⁠^{[2]IETF, RFC Editor (February 2024). RFC 9421: HTTP Message Signatures} Neither decides what your store does with that visit. That is still up to your robots.txt, your AI bot blocking rules and your content negotiation, which decides which version it receives.

To keep going, the entry on caching and CDN for bots explains what happens to those requests before they reach your server, and the crawling and indexing reading path orders the rest of the terms.