# Bingbot: What It Is and How to Verify a Visit Is from Bing > What Bingbot is, the crawler behind Bing and Copilot: its user agents, how to control it in robots.txt and how to check a visit is genuine. URL: https://clione.ai/glossary/bingbot/ Lang: en Bingbot is not [Bing Webmaster Tools](/glossary/bing-webmaster-tools/), which is the dashboard where you see how Bing treats you, or [Copilot](/glossary/copilot/), which is the assistant: it is the [crawler](/glossary/web-crawler/) that visits your store and takes the content away. It matters more than it tends to seem, because **Bing says its search and Copilot rely on the same foundation of crawling, [indexing](/glossary/indexing/) and ranking**: what Bingbot cannot read reaches neither of them.[^bing-guidelines] And it matters for a more prosaic reason: anyone can use its name in a request, and what you see in your server logs is not always Bing. ## How it works Bing has several crawlers, and Bingbot is the main one: according to Bing, it **handles most of its daily crawling**.[^bing-crawlers] The others have specific jobs: | Crawler | What Bing uses it for | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | Bingbot | Standard search crawling.[^bing-crawlers] | | AdIdxBot | Crawls Bing Ads ads and the sites they lead to, for quality control.[^bing-crawlers] | | BingPreview | Generates page snapshots for Bing.[^bing-crawlers] | | MicrosoftPreview | Generates page snapshots for other Microsoft products.[^bing-crawlers] | | BingVideoPreview | Generates video previews, in Bing only.[^bing-crawlers] | Bingbot presents itself with several [user agent](/glossary/user-agent/) strings, a desktop one, a mobile one and a short one, and all of them carry `bingbot/2.0`. Where it says `W.X.Y.Z` goes the version of Microsoft Edge it is using at that moment: **Bing updates its rendering engine to the latest stable version of Edge**, so a store that paints its content with JavaScript is read with a modern browser.[^bing-crawlers] That is not a blank check: its guidelines ask you not to hide critical content behind [client-side rendering](/glossary/javascript-rendering/) and warn that what cannot be rendered reliably may go unindexed.[^bing-guidelines] ```text Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) ``` A detail worth knowing when reading logs: in Bing's table, **BingPreview uses the same strings with `bingbot/2.0`** as Bingbot.[^bing-crawlers] From the user agent you will not always know which of the two visited you. **How it is controlled.** Bing offers two routes: the [robots.txt](/glossary/robots-txt/) file and, in Bing Webmaster Tools, a crawl control tool that regulates the rate by hour.[^bing-crawlers] In `robots.txt` Bing supports rules specific to its crawler, such as `Crawl-delay`, in a separate `User-agent: Bingbot` group. With a condition that tends to catch people off guard: **when Bingbot finds a group of its own, it ignores the rules of the general group**, so you have to repeat them all in its own. Bing therefore discourages per-bot groups unless they are needed. And changes are not immediate: search engines keep a cached copy of `robots.txt` for at least a few hours.[^bing-robots] **What Copilot can use.** Crawling is one thing and the use of content is another. Bing supports _meta robots_ tags (or the `X-Robots-Tag` header) that you can aim only at it with `name="bingbot"`. Three matter to a store:[^bing-meta] - [`noindex`](/glossary/noindex/): the page does not enter Bing's index. Its guidelines add that it also does not appear in Copilot or in the [grounding](/glossary/grounding/) results of its API.[^bing-guidelines] - `noarchive`: Bing does not keep a copy, **does not link it in Copilot** and does not use it to train Microsoft's generative AI foundation models.[^bing-meta] - `nocache`: Copilot can only show the URL, the title and the snippet. If a page carries `nocache` and `noarchive`, Bing applies `nocache`.[^bing-meta] Bing sums it up like this in its guidelines: `noarchive` keeps the content from being used in Copilot answers and in grounding results, and `nocache` reduces the depth of citations, so **to be cited in detail, don't put `nocache` on the pages you want to see in Copilot**.[^bing-guidelines] ## In a store **Fictitious example.** A store wants Bingbot to go slower and adds a group of its own to its `robots.txt`, below the rules it already had for everyone: ```text User-agent: * Disallow: /checkout/ Disallow: /cart/ User-agent: Bingbot Crawl-delay: 1 ``` The result is not what it was after. Because Bingbot now has its own group, it stops reading the general one and **can crawl `/checkout/` and `/cart/`**, which the store thought were closed.[^bing-robots] The correct version repeats the rules in Bingbot's group: ```text User-agent: * Disallow: /checkout/ Disallow: /cart/ User-agent: Bingbot Crawl-delay: 1 Disallow: /checkout/ Disallow: /cart/ ``` And if what the store wanted was for Bing to visit less at certain hours, the crawl control tool in Bing Webmaster Tools does that without touching the file.[^bing-crawlers] Believing a visit is from Bing because the user agent says `bingbot`. Bing warns about it: **user agent strings are easy to fake**, and not every request that carries them comes from one of its crawlers.[^bing-crawlers] Blocking or allowing by user agent in the firewall, or drawing conclusions from the logs without verifying the IPs, mixes Bing in with whoever is posing as it. ## How to check it 1. **Search for `bingbot` in the logs** of your server or your CDN and note the IPs of the visits. 2. **Verify an IP with two DNS lookups.** The reverse one has to return a name ending in `search.msn.com`, and the forward one on that name has to return the same IP. On Windows `nslookup` works and on Linux, `host`; you can also use Bing's Verify Bingbot tool.[^bing-verify] 3. **Or compare it with the official list.** Bing publishes Bingbot's IP ranges in `bingbot.json`[^bingbot-ips] and asks you to refresh that list daily, because it can change at any time; it also discourages hard-coding IPs in your configuration.[^bing-verify] 4. **Read your `robots.txt`.** If it has a `User-agent: Bingbot` group, check that it repeats all the rules of the general group.[^bing-robots] 5. **Look at the _meta robots_ tags and the `X-Robots-Tag` header of your [product pages](/glossary/product-page/).** A `noarchive` or a `nocache` inherited from a template decides how much of your catalog Copilot can use.[^bing-meta] ## Related concept: Googlebot [Googlebot](/glossary/googlebot/) does for Google what Bingbot does for Bing, and most `robots.txt` rules work for both. The differences are in the details. Google **does not support `Crawl-delay`**: its documentation lists it among the fields it does not interpret,[^google-robots] while Bing gives it as an example of a rule for its crawler.[^bing-robots] And control over AI use goes down different roads. At Bing it is the `noarchive` and `nocache` tags on the page itself.[^bing-meta] At Google, to limit training and grounding in some of its other systems, Google points to [Google-Extended](/glossary/google-extended/).[^google-ai-features] Each is verified with its own domains and its own IP lists. To go further, the [crawling and indexing reading path](/learn/crawling-and-indexing/) orders the rest of the terms, and the entry on [IndexNow](/glossary/indexnow/) explains how to tell Bing about each change instead of waiting for Bingbot to come back.