/ GUIDE — 5 min read

Generate an XML sitemap from a crawl

Build a sitemap that only lists pages search engines should index — from what Nosy Bat actually crawled, not from a guess.

What a sitemap should and shouldn't contain

A sitemap is a list of URLs you want indexed, with optional hints about when each changed. Search engines treat it as a suggestion, not a command, so a sitemap full of redirects, noindexed pages and duplicates does more harm than good: it wastes crawl budget and tells the engine your signals can't be trusted.

Nosy Bat builds the sitemap from the crawl, so it only contains pages that are indexable today: HTML, status 200, no noindex, not canonicalised to another URL. Pages robots.txt keeps out are never crawled, so they never get in either.

Of 1,240 crawled URLs, 980 are indexable — and those 980, no more, go in the sitemap. Redirects, noindexed and canonicalised pages and non-HTML files drop out.

1. Crawl the site

Any completed crawl in Spider (follow links) mode works; list-mode crawls don't offer the generator. If part of the site shouldn't be listed (a staging folder, internal tools), exclude it before you start with Advanced options → Configuration → Never crawl URLs matching, or leave it out in the generator's options below.

2. Open the generator

On the crawl's results page, open Crawl details and then Generate XML sitemap. What it includes is fixed — indexable pages only — and the rest is yours to set:

lastmod — Crawl date (the default), the page's Last-Modified header when the server sends one, or Leave out. Use the header if your CMS sets it correctly; otherwise leave it out. A wrong lastmod is worse than no lastmod.
changefreq and priority — Leave out, or a fixed value; priority can also go By depth (1.0 for the home page). Most engines ignore both, so leaving them out is fine.
Leave out URLs matching (one per line) — paths to exclude even if they are indexable.
Add an image sitemap — a second file listing the images found on each page.

3. Download and place the files

Download .zip gives you sitemap.xml — or, past 50,000 URLs, sitemap-1.xml, sitemap-2.xml… with a sitemap_index.xml that points at each one. Upload them to the root of the site, next to robots.txt, so the sitemap URL is https://your-site.com/sitemap.xml (or the index's).

Past 50,000 URLs the generator splits the sitemap into numbered files and adds an index that points at each one. You submit the index.

Sites change; sitemaps go stale. A scheduled crawl means there is always a fresh crawl to build the file from. See plans →

4. Tell the engines

Add one line to robots.txt:

Sitemap: https://your-site.com/sitemap.xml

Then submit the same URL in Google Search Console under Sitemaps. On the next crawl, Nosy Bat's Sitemaps family checks the result: No XML sitemap found, Sitemap not declared in robots.txt, Non-indexable URL in the sitemap, Redirect in the sitemap and Error page in the sitemap should all come back empty.

The file sits at the site root, robots.txt names it on a Sitemap: line, and Search Console has it submitted.

5. Keep it current

Regenerate after any structural change — new sections, removed pages, a migration. If you crawl on a schedule, every completed run is a fresh crawl to build from: open its Crawl details, set the options and download. The options are not saved with the schedule, so note the ones you chose.