What is IndexNow and what is it for?

IndexNow is not a Google tool or a plugin that pushes you up the rankings: it is a notice. Instead of waiting for a crawler to come back to your site and notice that something has changed, your site tells it. According to the initiative itself, without that notice search engines can take anywhere from days to weeks to discover that content has changed.⁠^{[1]IndexNow.org. IndexNow} For a store, what changes often is exactly what matters most to keep current: a new product, a marked-down price, a product page that sells out or is retired. And there is a new reason: Bing recommends IndexNow to keep information fresh in search and in AI experiences, Copilot among them.⁠^{[2]Microsoft, Webmaster Tools blog. Introducing AI Performance in Bing Webmaster Tools Public Preview}

How it works

The protocol has three pieces: a key, a file that proves it and the request that announces the change.⁠^{[3]IndexNow.org. Documentation}

The key. It is a string of 8 to 128 characters that can contain only letters, numbers and hyphens. To prove the site is yours, you publish a UTF-8 text file named after the key, {key}.txt, at the root of the domain; when you submit URLs, the search engine reads that file to check you are the owner. It can also be stored at another path and pointed to with the keyLocation parameter, but then it is only valid for URLs that start with that path.⁠^{[3]IndexNow.org. Documentation}

The notice. For one URL, a GET request to the search engine's address with two parameters, url and key, is enough. For several, a POST request with a JSON body carrying the host, the key and the list of URLs: up to 10,000 URLs per submission.⁠^{[3]IndexNow.org. Documentation}

The response. A 200 means the URL was received; a 202, that it was received and the key is pending validation. A 400 is an invalid format; a 403, that the key is not valid or is not in the file; a 422, that the URLs do not belong to the host or the key does not meet the format; and a 429, too many requests, with possible suspicion of spam.⁠^{[3]IndexNow.org. Documentation}

Who gets it. A notice sent to any participating search engine is shared with all the others.⁠^{[3]IndexNow.org. Documentation} The initiative's list includes Amazon, Bing, Naver, Seznam.cz, Yandex and Yep, plus a global submission endpoint.⁠^{[4]IndexNow.org. FAQ} Google is not on it. For Google, the equivalent route is the XML sitemap: it asks you to keep it up to date and, if the content changes, to include the <lastmod> tag.⁠^{[5]Google Search Central. Optimize your crawl budget}

What to send. The URLs that have been added, updated or deleted: new products, changed descriptions, price or availability changes, user content and deleted pages. Also redirected URLs and those that return a 404 or a 410, so the search engine updates its index. What you should not do is send the same URL many times a day with no content changes; if you need to repeat, the initiative asks you to wait at least five minutes. Daily limits are not public: each search engine sets its own per site.⁠^{[4]IndexNow.org. FAQ}

What it does not do. The protocol says it in so many words: the 200 only indicates that the search engine received the URL.⁠^{[3]IndexNow.org. Documentation} Bing repeats it: IndexNow makes sure search engines know about your changes, but it does not guarantee that the pages will be crawled or indexed.⁠^{[6]Microsoft Bing. IndexNow: get started} What it does aim for is that the search engine prioritizes those URLs and needs less crawling of its own to discover what is new.⁠^{[1]IndexNow.org. IndexNow}

On a store

A store marks down two products and retires a third, which now returns a 404. It announces all three at once with a single request. This is a fictional example: the domain, the key and the URLs are invented.

POST /indexnow HTTP/1.1
Host: <search engine>
Content-Type: application/json; charset=utf-8

{
  "host": "example-store.com",
  "key": "4f6a1c2e9b7d40a8b3e5c1d2f0a9b8c7",
  "keyLocation": "https://example-store.com/4f6a1c2e9b7d40a8b3e5c1d2f0a9b8c7.txt",
  "urlList": [
    "https://example-store.com/products/urban-backpack-20",
    "https://example-store.com/products/travel-backpack-40",
    "https://example-store.com/products/backpack-2024-edition"
  ]
}

The third URL is on the list on purpose: a retired product page is also a change, and announcing it helps the search engine update its index.⁠^{[4]IndexNow.org. FAQ} If the response is a 200, all you know is that the notice arrived.

You almost never need to write this request by hand. Bing cites several content management platforms with IndexNow built in, among them Wix, Duda, GoDaddy and Shopify, and one CDN, Cloudflare, through its Crawler Hints feature.⁠^{[6]Microsoft Bing. IndexNow: get started} Cloudflare turns it on from the dashboard and uses responses that were not in its cache as a hint that the content has changed, to send the notice.⁠^{[7]Cloudflare Docs. Crawler Hints} On Shopify, the mention is Bing's: we have not found in Shopify's help how it does it, so check in Bing Webmaster Tools before installing an app that does the same.

How to check it

1. See whether you already have it. Check whether your platform or your CDN is among those Bing cites as having IndexNow built in and, if you use Cloudflare, whether Crawler Hints is turned on in the dashboard.⁠^{[6]Microsoft Bing. IndexNow: get started}⁠^{[7]Cloudflare Docs. Crawler Hints}
2. Open the key file at yourdomain.com/{key}.txt. It has to respond and contain the same key the notices use, or you will get a 403.⁠^{[3]IndexNow.org. Documentation}
3. Send a test URL and read the code. A 200 or a 202 is correct; a 422 usually means URLs from another host, and a 429 that you are notifying too much.⁠^{[3]IndexNow.org. Documentation}
4. Check receipt in Bing Webmaster Tools. Bing says that is where you verify whether search engines have received your URLs.⁠^{[6]Microsoft Bing. IndexNow: get started}
5. Do not forget Google. Keep the sitemap with a <lastmod> that changes only when the content changes.⁠^{[5]Google Search Central. Optimize your crawl budget}

Related concept: XML sitemap

Both exist so a search engine learns your URLs, but they work in opposite directions. The sitemap is a list the search engine goes to fetch when it decides to: it describes which URLs exist and when they changed. IndexNow is a notice the site sends at the moment of the change, URL by URL. The sitemap waits to be read; IndexNow knocks on the door. They complement each other: the sitemap remains the route for Google, and IndexNow notices reach every search engine that uses it. Neither guarantees crawling, and neither on its own saves the crawl budget that is lost on useless URLs.

To keep going, the crawling and indexing reading path puts the rest of the terms in order.