What is Bingbot, Bing's crawler?

Bingbot is not Bing Webmaster Tools, which is the dashboard where you see how Bing treats you, or Copilot, which is the assistant: it is the crawler that visits your store and takes the content away. It matters more than it tends to seem, because Bing says its search and Copilot rely on the same foundation of crawling, indexing and ranking: what Bingbot cannot read reaches neither of them.⁠^{[1]Bing Webmaster Tools, help. Bing Webmaster Guidelines} And it matters for a more prosaic reason: anyone can use its name in a request, and what you see in your server logs is not always Bing.

How it works

Bing has several crawlers, and Bingbot is the main one: according to Bing, it handles most of its daily crawling.⁠^{[2]Bing Webmaster Tools, help. Which Crawlers Does Bing Use?} The others have specific jobs:

Bingbot presents itself with several user agent strings, a desktop one, a mobile one and a short one, and all of them carry bingbot/2.0. Where it says W.X.Y.Z goes the version of Microsoft Edge it is using at that moment: Bing updates its rendering engine to the latest stable version of Edge, so a store that paints its content with JavaScript is read with a modern browser.⁠^{[2]Bing Webmaster Tools, help. Which Crawlers Does Bing Use?} That is not a blank check: its guidelines ask you not to hide critical content behind client-side rendering and warn that what cannot be rendered reliably may go unindexed.⁠^{[1]Bing Webmaster Tools, help. Bing Webmaster Guidelines}

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

A detail worth knowing when reading logs: in Bing's table, BingPreview uses the same strings with bingbot/2.0 as Bingbot.⁠^{[2]Bing Webmaster Tools, help. Which Crawlers Does Bing Use?} From the user agent you will not always know which of the two visited you.

How it is controlled. Bing offers two routes: the robots.txt file and, in Bing Webmaster Tools, a crawl control tool that regulates the rate by hour.⁠^{[2]Bing Webmaster Tools, help. Which Crawlers Does Bing Use?} In robots.txt Bing supports rules specific to its crawler, such as Crawl-delay, in a separate User-agent: Bingbot group. With a condition that tends to catch people off guard: when Bingbot finds a group of its own, it ignores the rules of the general group, so you have to repeat them all in its own. Bing therefore discourages per-bot groups unless they are needed. And changes are not immediate: search engines keep a cached copy of robots.txt for at least a few hours.⁠^{[3]Bing Webmaster Tools, help. How to Create a robots.txt File}

What Copilot can use. Crawling is one thing and the use of content is another. Bing supports meta robots tags (or the X-Robots-Tag header) that you can aim only at it with name="bingbot". Three matter to a store:⁠^{[4]Bing Webmaster Tools, help. Robots meta tags and attributes that Bing supports}

• noindex: the page does not enter Bing's index. Its guidelines add that it also does not appear in Copilot or in the grounding results of its API.⁠^{[1]Bing Webmaster Tools, help. Bing Webmaster Guidelines}
• noarchive: Bing does not keep a copy, does not link it in Copilot and does not use it to train Microsoft's generative AI foundation models.⁠^{[4]Bing Webmaster Tools, help. Robots meta tags and attributes that Bing supports}
• nocache: Copilot can only show the URL, the title and the snippet. If a page carries nocache and noarchive, Bing applies nocache.⁠^{[4]Bing Webmaster Tools, help. Robots meta tags and attributes that Bing supports}

Bing sums it up like this in its guidelines: noarchive keeps the content from being used in Copilot answers and in grounding results, and nocache reduces the depth of citations, so to be cited in detail, don't put nocache on the pages you want to see in Copilot.⁠^{[1]Bing Webmaster Tools, help. Bing Webmaster Guidelines}

In a store

Fictitious example. A store wants Bingbot to go slower and adds a group of its own to its robots.txt, below the rules it already had for everyone:

User-agent: *
Disallow: /checkout/
Disallow: /cart/

User-agent: Bingbot
Crawl-delay: 1

The result is not what it was after. Because Bingbot now has its own group, it stops reading the general one and can crawl /checkout/ and /cart/, which the store thought were closed.⁠^{[3]Bing Webmaster Tools, help. How to Create a robots.txt File} The correct version repeats the rules in Bingbot's group:

User-agent: *
Disallow: /checkout/
Disallow: /cart/

User-agent: Bingbot
Crawl-delay: 1
Disallow: /checkout/
Disallow: /cart/

And if what the store wanted was for Bing to visit less at certain hours, the crawl control tool in Bing Webmaster Tools does that without touching the file.⁠^{[2]Bing Webmaster Tools, help. Which Crawlers Does Bing Use?}

How to check it

1. Search for bingbot in the logs of your server or your CDN and note the IPs of the visits.
2. Verify an IP with two DNS lookups. The reverse one has to return a name ending in search.msn.com, and the forward one on that name has to return the same IP. On Windows nslookup works and on Linux, host; you can also use Bing's Verify Bingbot tool.⁠^{[5]Bing Webmaster Tools, help. How to Verify Bingbot}
3. Or compare it with the official list. Bing publishes Bingbot's IP ranges in bingbot.json⁠^{[6]Microsoft Bing. bingbot.json (Bingbot IP ranges)} and asks you to refresh that list daily, because it can change at any time; it also discourages hard-coding IPs in your configuration.⁠^{[5]Bing Webmaster Tools, help. How to Verify Bingbot}
4. Read your robots.txt. If it has a User-agent: Bingbot group, check that it repeats all the rules of the general group.⁠^{[3]Bing Webmaster Tools, help. How to Create a robots.txt File}
5. Look at the meta robots tags and the X-Robots-Tag header of your product pages. A noarchive or a nocache inherited from a template decides how much of your catalog Copilot can use.⁠^{[4]Bing Webmaster Tools, help. Robots meta tags and attributes that Bing supports}

Related concept: Googlebot

Googlebot does for Google what Bingbot does for Bing, and most robots.txt rules work for both. The differences are in the details. Google does not support Crawl-delay: its documentation lists it among the fields it does not interpret,⁠^{[7]Google Search Central. How Google interprets the robots.txt specification} while Bing gives it as an example of a rule for its crawler.⁠^{[3]Bing Webmaster Tools, help. How to Create a robots.txt File} And control over AI use goes down different roads. At Bing it is the noarchive and nocache tags on the page itself.⁠^{[4]Bing Webmaster Tools, help. Robots meta tags and attributes that Bing supports} At Google, to limit training and grounding in some of its other systems, Google points to Google-Extended.⁠^{[8]Google Search Central. AI features and your website} Each is verified with its own domains and its own IP lists.

To go further, the crawling and indexing reading path orders the rest of the terms, and the entry on IndexNow explains how to tell Bing about each change instead of waiting for Bingbot to come back.