What is a user agent?

The user agent is not an AI agent or a program you have to install: it is the line of text with which whoever requests a page from your server introduces itself. A customer's browser sends it, Googlebot sends it and every AI company's bot sends it. It matters to a store for two reasons. It is the first thing you look at to find out who visited a product page, in the server logs or in your CDN's stats. And it is the name that the rules in robots.txt are addressed to. What it does not do is prove anything, because the one making the request writes it.

How it works

The HTTP standard, RFC 9110, defines the User-Agent header as information about the program that originates the request. It says a client should send it in every request and explains what servers use it for: to scope compatibility problems, to work around or tailor responses to the limitations of a particular program, and for statistics on browsers and operating systems.⁠^{[1]IETF (Fielding, Nottingham and Reschke). RFC 9110: HTTP Semantics, section 10.1.5 (User-Agent)} The format is a list of products, each with its name and, optionally, its version after a slash, followed by comments in parentheses, ordered from most to least significant. The standard's example is CERN-LineMode/2.15 libwww/2.17b3.⁠^{[1]IETF (Fielding, Nottingham and Reschke). RFC 9110: HTTP Semantics, section 10.1.5 (User-Agent)}

In practice, browser strings are longer and stranger. Almost every browser starts with Mozilla/5.0, a fragment that today only says the browser is compatible with Mozilla and is sent for historical reasons.⁠^{[2]MDN Web Docs (Mozilla). User-Agent header} The standard itself discourages using other programs' names to claim compatibility with them, because that defeats the purpose of the header.⁠^{[1]IETF (Fielding, Nottingham and Reschke). RFC 9110: HTTP Semantics, section 10.1.5 (User-Agent)} They do it anyway: that is why a bot like OAI-SearchBot sends a string that starts like Chrome on a Mac.⁠^{[3]OpenAI. Overview of OpenAI Crawlers}

It says less and less about the visitor. The RFC warns that an overly detailed string helps identify a person without their wanting it, which is known as fingerprinting.⁠^{[1]IETF (Fielding, Nottingham and Reschke). RFC 9110: HTTP Semantics, section 10.1.5 (User-Agent)} Chrome responded by reducing its own: since Chrome 101 the version arrives as 0.0.0 after the major number, since Chrome 107 the desktop string started to be trimmed and since Chrome 110 the Android one, which no longer includes the phone model but a fixed K. The reduction was complete in Chrome 113.⁠^{[4]The Chromium Projects. User-Agent Reduction} If a server needs more detail, it can ask for it with client hints, Sec-CH-UA headers that the browser sends only when asked.⁠^{[2]MDN Web Docs (Mozilla). User-Agent header}

Bots have two names. Google distinguishes between the string that appears in the HTTP request and the short name written in the User-agent: line of robots.txt. For Googlebot Smartphone, the string is long and changes with every Chrome version; in robots.txt, Googlebot is enough.⁠^{[5]Google Crawling Infrastructure. Google's common crawlers} OpenAI adds a useful detail: when OAI-SearchBot or GPTBot request robots.txt itself, they may include robots.txt in the string so those requests can be told apart in the logs.⁠^{[3]OpenAI. Overview of OpenAI Crawlers}

And the name can be forged. Google says so in its list of crawlers: the user agent string can be spoofed.⁠^{[5]Google Crawling Infrastructure. Google's common crawlers} That is why its crawlers identify themselves in three ways at once: the User-Agent header, the source IP and the name returned by a reverse DNS lookup of that IP.⁠^{[6]Google Crawling Infrastructure. Google crawler (user agent) overview}

In a store

Four requests to the same product page, with the strings documented by Chromium, Google and OpenAI. The store and the order are a fictional example; W.X.Y.Z is the Chrome version, which changes:

Look at the second row: Googlebot's string includes "Android" and "Chrome". A filter that looks for "Chrome" or "Android" counts it as a customer. What identifies the bot is its own name inside the string, and what proves it is that bot is the check in the last column.

How to check it

1. Look at the user agent field in your logs. On the server or the CDN, every access line carries it at the end. Separate the requests from the bots you care about by their own name, not by "Chrome" or "Mozilla".
2. Verify bots by IP. For Google, run the reverse and forward DNS lookup, or compare the IP with its published ranges;⁠^{[6]Google Crawling Infrastructure. Google crawler (user agent) overview} OpenAI publishes a JSON file of IPs for each agent.⁠^{[3]OpenAI. Overview of OpenAI Crawlers}
3. Request your product page as someone else. In Chrome, the Network conditions panel in developer tools lets you replace the string: it does not change how the browser works, it changes what the server hands it.⁠^{[8]Chrome for Developers. Network conditions: Override the user agent string} From the terminal, curl -A "string" https://yourdomain.com/products/... does the same.
4. Compare what each one receives. If the title, the price or the structured data change depending on the string, find out what causes it before it affects the page's indexing.
5. Read bot dashboards with care. On the free plan, Cloudflare's AI Crawl Control identifies AI crawlers by their user agent string, so it only detects the ones that identify themselves.⁠^{[9]Cloudflare Docs, AI Crawl Control. Manage AI crawlers}

Related concept: crawler

A crawler is a program that walks the web; the user agent is the label it presents on every request. Every crawler has one, but so does your browser, your customer's and any script that requests a URL. Reading the user agent tells you who the visitor says it is; verifying its IP tells you whether it is. To go a step beyond the name, some agents sign their requests with Web Bot Auth, and that signature is what Cloudflare's pay per crawl, for example, requires.

To keep going, the crawling and indexing reading path orders the rest of the terms, and the Googlebot entry details how to verify Google's crawler step by step.