What is content negotiation in HTTP?

Content negotiation has nothing to do with haggling over a price, or with an agent bargaining on your behalf. It is a piece of HTTP, the protocol your server answers every visit with, and it settles one very specific question: when a URL has several versions, which one goes to whom. It matters more to a store than it did a few years ago because browsers are no longer the only visitors. An AI agent can ask for the same product page in Markdown, a crawler can ask for it without saying which language it wants, and between the two and your server there is usually a cache keeping copies. If negotiation is done badly, each of them can end up seeing someone else's version.

How it works

The standard is RFC 9110, published in June 2022. Its section 12 starts from an observation: a server often has several ways to represent the same information (formats, languages, encodings), and whoever asks for it has different capabilities and preferences, so HTTP offers mechanisms to negotiate which one is delivered.⁠^{[1]IETF, RFC Editor (June 2022). RFC 9110: HTTP Semantics (section 12, Content Negotiation)} It defines three patterns:⁠^{[1]IETF, RFC Editor (June 2022). RFC 9110: HTTP Semantics (section 12, Content Negotiation)}

• Proactive negotiation (server-driven): the client sends its preferences and the server chooses. To decide, it can use the negotiation headers and also implicit traits of the request, such as the network address or parts of the User-Agent field.
• Reactive negotiation (agent-driven): the server returns a list of alternatives and the client chooses, with a second request.
• Request content negotiation: the server says in its response which formats it prefers to receive in later requests.

Proactive is the one you see on the web. Four headers carry it: Accept (the format), Accept-Charset (the character set), Accept-Encoding (the compression) and Accept-Language (the language). Each preference can carry a q weight between 0 and 1, where 0 means "not acceptable" and, with no weight, the value is 1. A request without one of those headers means the client has no preference on that dimension. And if no version fits, the server can answer 406 (Not Acceptable) or ignore the header and serve the usual one.⁠^{[1]IETF, RFC Editor (June 2022). RFC 9110: HTTP Semantics (section 12, Content Negotiation)}

The standard itself does not pitch it as free. It lists its drawbacks: the server can never fully know what is best for each client, describing capabilities on every request is inefficient and can affect privacy, it complicates the server and it limits how much shared caches can reuse responses.⁠^{[1]IETF, RFC Editor (June 2022). RFC 9110: HTTP Semantics (section 12, Content Negotiation)}

The Vary header

That last drawback is handled with Vary. It is a response header that lists which parts of the request may have influenced the chosen version, and it has two jobs: to tell caches they cannot use that response for another request unless the listed headers match, and to tell the client that other versions exist. The RFC asks you to send it on any cacheable response when the content has been adapted to those headers, for example when picking the language from Accept-Language.⁠^{[1]IETF, RFC Editor (June 2022). RFC 9110: HTTP Semantics (section 12, Content Negotiation)} The caching standard finishes the thought: a cache cannot reuse a stored response without revalidating it if the headers Vary names do not match those of the new request.⁠^{[2]IETF, RFC Editor (June 2022). RFC 9111: HTTP Caching} That is why negotiation and caching and CDNs for bots always go together.

In a store

The most common case in 2026 is format. Cloudflare offers a feature, Markdown for Agents, that does exactly this: when a client adds Accept: text/markdown to the request, it asks the origin for the HTML version, converts it to Markdown and serves it with Content-Type: text/markdown; charset=utf-8, with Accept included in Vary so caches store the two versions separately. It is available on the Pro, Business and Enterprise plans.⁠^{[3]Cloudflare, documentation. Markdown for Agents} The format itself has its own entry, Markdown for agents.

Here is how the exchange would look in a store, as a made-up example. The agent prefers Markdown (weight 1, by default) and accepts HTML as a second choice (weight 0.8):

GET /products/stainless-moka-6 HTTP/1.1
Host: coffee-makers-example.com
Accept: text/markdown, text/html;q=0.8

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept

The line that matters is the last one. Without Vary: Accept, a shared cache that stored this response could hand it to the next browser that asks for the same URL.⁠^{[2]IETF, RFC Editor (June 2022). RFC 9111: HTTP Caching}

The other case is language, and there the risk comes from the crawler. Google says Googlebot makes its requests without the Accept-Language header and that its default IP addresses appear to be in the United States, although it also crawls from elsewhere. If your store picks the language from that header, Google warns that it may not crawl, index or rank all of your content in every language, and it recommends separate URLs per language, annotated with hreflang.⁠^{[4]Google Search Central. How Google crawls locale-adaptive pages} Under the RFC's rule for absent headers, a request without Accept-Language expresses no preference,⁠^{[1]IETF, RFC Editor (June 2022). RFC 9110: HTTP Semantics (section 12, Content Negotiation)} so Googlebot will see the default version and only that one.

How to check it

1. Request your product page as Markdown with curl -sI -H "Accept: text/markdown" https://your-domain/products/your-product and look at Content-Type. If it is still text/html, your server ignores that preference, which is a valid answer under the RFC.⁠^{[1]IETF, RFC Editor (June 2022). RFC 9110: HTTP Semantics (section 12, Content Negotiation)}
2. Check Vary every time the format or language changes. If the response changes with Accept or Accept-Language and Vary does not name that header, a shared cache may serve the wrong version.⁠^{[2]IETF, RFC Editor (June 2022). RFC 9111: HTTP Caching}
3. Request the page without Accept-Language. That is what Googlebot does, and what you get back is what it can crawl.⁠^{[4]Google Search Central. How Google crawls locale-adaptive pages}
4. Compare price and availability across all versions. If they do not match, the problem is no longer one of format.

Related concept: user agent

The user agent is the name the client introduces itself with, not a preference. The RFC allows proactive negotiation to use parts of that field as an implicit signal,⁠^{[1]IETF, RFC Editor (June 2022). RFC 9110: HTTP Semantics (section 12, Content Negotiation)} but they are different things: Accept says what the client wants; the user agent says who it claims to be, and anyone can write it. Choosing the version by user agent is guessing; choosing it by Accept is answering what was asked for. To verify who is really calling you need something else, such as Web Bot Auth.

To go further, the reading path on crawling and indexing orders the rest of the pieces that decide what a bot sees in your store.