# Web Bot Auth: What It Is and How a Bot Signs Requests

> What Web Bot Auth is, the IETF draft for bots and AI agents to sign their HTTP requests, who already uses it and how a signature is verified.

URL: https://clione.ai/glossary/web-bot-auth/
Lang: en

Today a [bot](/glossary/web-crawler/) introduces itself with its [_user agent_](/glossary/user-agent/), a name it writes itself. That is why Google explains how to verify its [crawlers](/glossary/web-crawler/): for the case where someone accesses your site **pretending to be Google**.[^google-verify] Web Bot Auth is the proposal to close that gap with cryptography: the bot signs every request and your server, or your CDN, checks the signature. It matters to a store for a practical reason: the decision about who to let in is increasingly made by a firewall, and an agent that shops on behalf of a customer and a scraper disguised as one should not get the same treatment.

## How it works

The base is an already published standard, RFC 9421 (February 2024), which describes how to create, encode and verify digital signatures over parts of an HTTP message. It defines two headers: `Signature-Input`, which says which parts of the message the signature covers and with what parameters, and `Signature`, which carries the signature. Its goal is to **provide end-to-end integrity and authenticity** for those parts, even when the message passes through intermediaries.[^rfc9421]

Web Bot Auth is a profile of that standard for automated traffic. The working group's document, _HTTP Message Signatures for automated traffic_, by Thibault Meunier (Cloudflare) and Sandor Major (Google), describes a protocol for automated HTTP clients to identify themselves cryptographically, and adds three pieces:[^ietf-httpsig-bots]

- **The `Signature-Agent` header**, which points to an HTTPS URL where the signer's keys are.
- **A key directory** in JWKS format (_JSON Web Key Set_), published at `/.well-known/http-message-signatures-directory` on the signer's domain, served over HTTPS and only with a 200. It is a path in the [.well-known directory](/glossary/well-known/).
- **Rules for the signature**: the tag `tag="web-bot-auth"`, the signature has to cover at least the destination domain (`@authority` or `@target-uri`), and it carries a creation time, an expiry time (**a maximum of 24 hours** is recommended) and the key identifier. An optional `nonce` protects against reuse of a signed request.

To verify, whoever receives the request reads `Signature-Agent`, downloads the key directory, looks up the key by its identifier and checks the signature according to RFC 9421.[^ietf-httpsig-bots]

### Where it stands

The IETF created a working group, webbotauth, to standardize methods that cryptographically authenticate automated clients and give more information about who operates them. Its charter leaves out, among other things, authenticating an agent's end user, **tracking bots or assigning them reputation**, and telling bots that do not participate apart from people. The schedule called for the authentication specifications by April 30, 2026.[^ietf-wg-charter]

The working group document came later: it is version 00, from September 1, 2026, and expires on March 5, 2027. The two individual drafts that preceded it, the architecture one and the directory one, are listed as replaced, with their last version from March 2, 2026.[^ietf-wg-docs][^ietf-httpsig-bots] And it is an _Internet-Draft_: a working document valid for six months at most, which **can be updated, replaced or made obsolete at any time**.[^ietf-httpsig-bots]

## Who already uses it

- **Google.** It signs with Web Bot Auth part of the requests from Google-Agent, the agent that systems hosted on Google's infrastructure use to browse the web and take actions at a user's request. It marks it as experimental, because the specification is a draft that may change, and recommends that you **keep checking the IP, the reverse DNS and the _user agent_ too** while it extends signed traffic.[^google-wba] As a user-triggered fetcher, Google-Agent generally does not follow [robots.txt](/glossary/robots-txt/) rules.[^google-user-fetchers]
- **Cloudflare.** It uses Web Bot Auth as one of the methods to verify signed bots and agents, requires Ed25519 keys and lets the bot's operator register its key directory from the dashboard. According to its documentation, **the site owner does not have to do anything**: Cloudflare validates the signatures of registered bots.[^cloudflare-wba] Its documentation classifies verified bots by how they are operated: directly or through an intermediary.[^cloudflare-signed-agents]

## In a store

This is how a signed request from Google-Agent would reach the [product page](/glossary/product-page/) of a **fictional example** store. The structure follows the draft's example and the value of `Signature-Agent` is the one Google documents; the dates, the key identifier and the signature are made up:[^ietf-httpsig-bots][^google-wba]

```http
GET /products/moka-steel-6 HTTP/1.1
Host: coffee-makers.example
User-Agent: Mozilla/5.0 (X11; Linux x86_64) ... (compatible; Google-Agent)
Signature-Agent: g="https://agent.bot.goog"
Signature-Input: g=("@authority" "signature-agent";key="g");created=1791273600;expires=1791277200;keyid="example-key";tag="web-bot-auth"
Signature: g=:ZXhhbXBsZS1zaWduYXR1cmU=:
```

Line by line: `User-Agent` is what the bot says it is and proves nothing. `Signature-Agent` says where its keys are: `https://agent.bot.goog/.well-known/http-message-signatures-directory`.[^google-wba] `Signature-Input` says the signature covers the store's domain and the `Signature-Agent` header itself, valid for one hour. If someone copies the request to another site, the signature stops being valid, because it covered `coffee-makers.example`. If someone makes up the signature, they cannot generate it without the private key.


Treating Web Bot Auth as a finished standard and dropping the other checks. It is an IETF draft,[^ietf-httpsig-bots] Google only signs part of an agent's requests and asks you to **keep verifying by IP, reverse DNS and _user agent_**.[^google-wba] A bot that does not sign is not an impostor for that reason: the signature is one more proof, still partial.


## How to check it

1. **Ask your CDN or your platform whether it verifies Web Bot Auth.** If you are behind Cloudflare, Cloudflare does the verification with registered bots.[^cloudflare-wba]
2. **Look for the `Signature-Agent` header** in your server or CDN logs, if they keep request headers. Its value tells you which signer the request declares.
3. **Open a key directory**, for example Google's at `https://agent.bot.goog/.well-known/http-message-signatures-directory`, to see what a signer publishes.[^google-wba]
4. **Keep verifying with reverse DNS and IP ranges** for [Googlebot](/glossary/googlebot/) and Google's other crawlers.[^google-verify]
5. **Check the draft's status on the IETF Datatracker** before building anything custom: the current version is 00, from September 2026.[^ietf-wg-docs]

## Related concept: user agent

The _user agent_ is a statement; Web Bot Auth is proof. Both travel in the same request and answer the same question, who is calling, with different degrees of trust: the first is written by the client, the second can only be produced by whoever holds the key.[^rfc9421] Neither decides what your store does with that visit. That is still up to your `robots.txt`, your [AI bot blocking](/glossary/ai-bot-blocking/) rules and your [content negotiation](/glossary/content-negotiation/), which decides which version it receives.

To keep going, the entry on [caching and CDN for bots](/glossary/cdn-caching-for-bots/) explains what happens to those requests before they reach your server, and the crawling and [indexing](/glossary/indexing/) [reading path](/learn/crawling-and-indexing/) orders the rest of the terms.