# What Is Content Negotiation in HTTP? > What content negotiation is in HTTP, how a server picks between HTML, Markdown or languages with Accept and Vary, and what changes for bots. URL: https://clione.ai/glossary/content-negotiation/ Lang: en Content negotiation has nothing to do with haggling over a price, or with an agent bargaining on your behalf. It is a piece of HTTP, the protocol your server answers every visit with, and it settles one very specific question: **when a URL has several versions, which one goes to whom**. It matters more to a store than it did a few years ago because browsers are no longer the only visitors. An [AI agent](/glossary/ai-agent/) can ask for the same [product page](/glossary/product-page/) in Markdown, a [crawler](/glossary/web-crawler/) can ask for it without saying which language it wants, and between the two and your server there is usually a cache keeping copies. If negotiation is done badly, each of them can end up seeing someone else's version. ## How it works The standard is RFC 9110, published in June 2022. Its section 12 starts from an observation: a server often has several ways to represent the same information (formats, languages, encodings), and whoever asks for it has different capabilities and preferences, so HTTP offers mechanisms to negotiate which one is delivered.[^rfc9110] It defines three patterns:[^rfc9110] - **Proactive negotiation** (_server-driven_): the client sends its preferences and **the server chooses**. To decide, it can use the negotiation headers and also implicit traits of the request, such as the network address or parts of the [User-Agent](/glossary/user-agent/) field. - **Reactive negotiation** (_agent-driven_): the server returns a list of alternatives and **the client chooses**, with a second request. - **Request content negotiation**: the server says in its response which formats it prefers to receive in later requests. Proactive is the one you see on the web. Four headers carry it: `Accept` (the format), `Accept-Charset` (the character set), `Accept-Encoding` (the compression) and `Accept-Language` (the language). Each preference can carry a `q` weight between 0 and 1, where 0 means "not acceptable" and, with no weight, the value is 1. A request without one of those headers means the client has no preference on that dimension. And if no version fits, the server can answer 406 (_Not Acceptable_) or ignore the header and serve the usual one.[^rfc9110] The standard itself does not pitch it as free. It lists its drawbacks: the server can never fully know what is best for each client, describing capabilities on every request is inefficient and can affect privacy, it complicates the server and **it limits how much shared caches can reuse responses**.[^rfc9110] ### The Vary header That last drawback is handled with `Vary`. It is a response header that lists which parts of the request may have influenced the chosen version, and it has two jobs: to tell caches they cannot use that response for another request unless the listed headers match, and to tell the client that other versions exist. The RFC asks you to send it on any cacheable response when the content has been adapted to those headers, for example when picking the language from `Accept-Language`.[^rfc9110] The caching standard finishes the thought: a cache cannot reuse a stored response without revalidating it if the headers `Vary` names do not match those of the new request.[^rfc9111] That is why negotiation and [caching and CDNs for bots](/glossary/cdn-caching-for-bots/) always go together. ## In a store The most common case in 2026 is format. Cloudflare offers a feature, [Markdown for Agents](/glossary/markdown-for-agents/), that does exactly this: when a client adds `Accept: text/markdown` to the request, it asks the origin for the HTML version, converts it to Markdown and serves it with `Content-Type: text/markdown; charset=utf-8`, **with `Accept` included in `Vary`** so caches store the two versions separately. It is available on the Pro, Business and Enterprise plans.[^cloudflare-markdown] The format itself has its own entry, [Markdown for agents](/glossary/markdown-for-agents/). Here is how the exchange would look in a store, as a **made-up example**. The agent prefers Markdown (weight 1, by default) and accepts HTML as a second choice (weight 0.8): ```http GET /products/stainless-moka-6 HTTP/1.1 Host: coffee-makers-example.com Accept: text/markdown, text/html;q=0.8 HTTP/1.1 200 OK Content-Type: text/markdown; charset=utf-8 Vary: Accept ``` The line that matters is the last one. Without `Vary: Accept`, a shared cache that stored this response could hand it to the next browser that asks for the same URL.[^rfc9111] The other case is language, and there the risk comes from the crawler. Google says [Googlebot](/glossary/googlebot/) makes its requests **without the `Accept-Language` header** and that its default IP addresses appear to be in the United States, although it also crawls from elsewhere. If your store picks the language from that header, Google warns that it may not crawl, index or rank all of your content in every language, and it recommends separate URLs per language, annotated with [hreflang](/glossary/hreflang/).[^google-locale] Under the RFC's rule for absent headers, a request without `Accept-Language` expresses no preference,[^rfc9110] so Googlebot will see the default version and only that one. Calling it content negotiation when you serve a different page to bots. Negotiation delivers **another representation of the same resource**: the same product, the same price, in another format or language. The RFC makes clear that this consistency is entirely up to whoever generates the responses, not HTTP.[^rfc9110] And Google calls it _cloaking_ when content shown to people and to search engines differs with the intent to manipulate rankings, with a very literal example: inserting text or keywords only when the one asking for the page is a search engine.[^google-spam] ## How to check it 1. **Request your product page as Markdown** with `curl -sI -H "Accept: text/markdown" https://your-domain/products/your-product` and look at `Content-Type`. If it is still `text/html`, your server ignores that preference, which is a valid answer under the RFC.[^rfc9110] 2. **Check `Vary` every time the format or language changes.** If the response changes with `Accept` or `Accept-Language` and `Vary` does not name that header, a shared cache may serve the wrong version.[^rfc9111] 3. **Request the page without `Accept-Language`.** That is what Googlebot does, and what you get back is what it can crawl.[^google-locale] 4. **Compare [price and availability](/glossary/price-and-availability/)** across all versions. If they do not match, the problem is no longer one of format. ## Related concept: user agent The [_user agent_](/glossary/user-agent/) is the name the client introduces itself with, not a preference. The RFC allows proactive negotiation to use parts of that field as an implicit signal,[^rfc9110] but they are different things: `Accept` says **what the client wants**; the _user agent_ says **who it claims to be**, and anyone can write it. Choosing the version by _user agent_ is guessing; choosing it by `Accept` is answering what was asked for. To verify who is really calling you need something else, such as [Web Bot Auth](/glossary/web-bot-auth/). To go further, the [reading path](/learn/crawling-and-indexing/) on crawling and [indexing](/glossary/indexing/) orders the rest of the pieces that decide what a bot sees in your store.