A sitemap should only list pages you actually want found

The reframe that fixes most sitemap problems is to stop treating it as "every page I have" and start treating it as "the pages I want you to index." Those are very different lists. Your site has all kinds of URLs that shouldn't be in the index — thin pages you've noindexed, old URLs that now redirect, utility pages that exist for function rather than search. None of those belong in a document whose entire purpose is to say "here's what's worth indexing." Including them isn't thoroughness; it's noise that dilutes the actual recommendation.

Once you see the sitemap as curated recommendations, the inclusion rule becomes obvious: a URL belongs in the sitemap only if it's a canonical page you genuinely want found. Everything else — noindexed, redirected, non-canonical — is excluded, not because it's shameful, but because recommending it would misrepresent what you actually want. The sitemap should be the clean list of your real answers, not a full accounting of every file that happens to respond.

A sitemap isn't an inventory of what exists. It's a list of what you want indexed — and everything you list that you don't want indexed is a recommendation working against you.

The specific damage of a sloppy sitemap is that it contradicts your other signals, and contradictions are expensive with search engines because they have to resolve them somehow. If your sitemap recommends a page while your page-level directives say don't index it, the crawler now has two conflicting instructions from you about the same URL. Best case, it wastes crawl attention figuring out which to believe. Worse case, the pattern of contradictions teaches it that your sitemap isn't a reliable guide, so it leans on it less — and now the recommendations you did mean carry less weight too.

Redirected URLs are the same story. If the sitemap points the crawler at a URL that just bounces it somewhere else, you've sent it on an errand that ends in a redirect — a small waste each time, and a signal that your sitemap doesn't reflect the current state of your site. A crawler that keeps finding your sitemap out of sync with reality treats the whole document as approximate. The trust you want it to place in your recommendations is exactly what a self-contradicting sitemap spends down.

The way to keep a sitemap honest is to stop maintaining it by hand and generate it from the same source of truth that drives your indexing decisions. If one place in your system knows which pages are noindexed and which URLs redirect where, then the sitemap should be produced from that same place — so it's incapable of disagreeing with it. A hand-maintained sitemap drifts out of sync the moment you noindex a page and forget to remove it from the list. A generated one can't, because it's reading the same facts.

Concretely, that means the sitemap-building step excludes anything the system knows is noindexed or redirected, automatically, every time it runs. There's no separate act of remembering to keep the two in agreement, because they're computed from the same underlying state. Consistency stops being a discipline I have to maintain and becomes a property of how the sitemap is made. The document and the directives can't contradict each other because they came from one source.

On my own site that source of truth is small enough to point at. Every essay and post is a row with a single boolean called noindex, and its whole job is spelled out in the migration that added it: "Emit meta robots noindex,follow and exclude from sitemap." One flag, two effects. When I set it on a page, that page starts serving a noindex directive to crawlers and drops out of the next sitemap in the very same act. There is no second switch to remember, so there is no way to flip one and forget the other. The contradiction I'm trying to avoid isn't something discipline has to catch — it's made unrepresentable.

That works because the sitemap builder is a database query, not a hand-kept list. The sub-sitemap for my long-form writing is nothing more than a filter for published essays that aren't noindexed and aren't redirected, minus anything dated in the future. Nothing in that file is remembered or maintained; the exclusion is the shape of the query itself.

# the /writing sub-sitemap is a query, so it cannot list what it excludes
BlogPost.objects.filter(
    status="published", post_type="writing",
    noindex=False, redirect_to="",
).exclude(
    published_date__gt=timezone.localdate()  # not public yet
)

Redirected pages fall out of the same single fact. The redirect_to field, when set, makes the old /blog/<slug>/ issue a 301 to the consolidated URL — and because the same query also filters on redirect_to being empty, setting that one value both performs the cannibalization consolidation and removes the stale URL from the sitemap. The redirect and the omission can't disagree, because they are the same value read twice.

The same principle scales from one page up to a whole section. When I decided to hide a large machine-generated area of the site, the honest way to do it was to make every signal agree at once. The web-server layer serves an X-Robots-Tag of noindex, nofollow on those bot-served pages, and the sitemap generator simply stops emitting them. It would have been a contradiction to keep listing thousands of those URLs in the sitemap while the response headers told the crawler not to index them. Instead both signals point the same direction, and the crawler never has to resolve a disagreement I created.

What makes that safe to do is that the whole hide is one reversible edit. In the generator, the builders for that section are still there, commented out with a note explaining why they're off. Uncomment them, drop the header, and the section returns to the index exactly as it was. Nothing about the pages' existence is denied; the sitemap just declines to recommend what the headers already decline to index — and the decision stays a switch I can flip back, not a demolition.

The generator holds back one more class of URL for the same reason: essays dated in the future. A scheduled post is real and already sitting in the database, but it isn't public yet, so the query excludes anything whose published date is past today. Advertising a URL the day before it's meant to exist is just one more way the sitemap would say something I don't mean. The rule underneath every one of these exclusions is identical — list only what is true right now, and let the query, not my memory, enforce it.

There's a broader principle here that goes past sitemaps. Every signal you send a search engine is a claim, and the value of your claims depends on their consistency and truth. A sitemap that only lists pages you actually want found, matching directives that actually reflect your intent, matching structured data that's actually accurate — that coherence is itself a signal. It says, in effect, this site's operator means what they say, and you can rely on their recommendations. That reliability compounds.

So keeping the sitemap honest isn't a fussy technicality; it's the same discipline that runs through everything — say exactly what's true, in a form that can be checked, and don't contradict yourself. A clean sitemap of genuine recommendations is a small thing that quietly makes every other signal you send more credible. The crawler is, in the end, deciding how much to trust you, and a sitemap that never lies to it is one more reason to.

This piece is part of my series on search, answer engines, and content quality. The anchor is AEO and GEO: optimizing for answer engines.