Your sitemap isn't enough — here's how to actually tell search engines about your pages

When I launched a new project, I did what most developers do: published the site, added a sitemap, submitted it to Google Search Console, and moved on. Job done.

What I didn't realise is that a sitemap isn't a notification — it's more like a phone book. Search engines might check it eventually. But they crawl on their own schedule, and for a new site or a newly published page, "eventually" can mean days or months. The sitemap just sits there waiting to be discovered.

There's a better way, and it's been around since 2021. Most developers have never heard of it.

What is IndexNow?

IndexNow is an open protocol that lets you push URLs directly to participating search engines the moment content is published or updated. Instead of waiting for a crawler to find your page, you send a single API request and the engine knows to crawl it immediately.

It was created by Microsoft and Yandex and is now supported by a growing list of search engines:

One request to api.indexnow.org reaches all of them simultaneously. You don't need to integrate with each engine separately.

The notable absence is Google. Google has its own URL Inspection API via Search Console, which works differently and requires OAuth credentials. But for every other major engine, IndexNow is the single endpoint.

How it works

The setup has two parts:

1. Prove you own the domain. Generate a random API key and host a plain text file at https://<your-host>/<your-key>.txt containing just the key. That's how IndexNow verifies you're not submitting URLs for someone else's site.

2. POST your URLs. Send a JSON payload to https://api.indexnow.org/indexnow:

{
  "host": "www.example.com",
  "key": "your-api-key",
  "keyLocation": "https://www.example.com/your-api-key.txt",
  "urlList": [
    "https://www.example.com/new-post",
    "https://www.example.com/updated-page"
  ]
}

You can submit up to 10,000 URLs in a single request. The engine responds with 200 and queues them for crawling.

Already built into PyNuxt

PyNuxt ships with IndexNow wired up out of the box. The key file is provisioned automatically via Terraform, and there's a script that fetches your sitemap, verifies each URL, and submits the clean set in a single batch — no setup beyond dropping in your key.