An XML sitemap that contains noindex URLs, 301 redirects, or pages blocked by robots.txt sends a contradictory signal to crawlers. Before considering submission or update frequency, we recommend treating the sitemap as a quality filter: only canonical, indexable, and accessible URLs deserve to be included.
XML Sitemap and Crawl Budget: The Often Underestimated Technical Link
Googlebot allocates a limited crawl budget to each site. Each URL present in the sitemap consumes a portion of this budget during crawling. A file that mixes useful pages with orphaned, duplicated, or redirected pages dilutes this budget on resources with no SEO value.
Related reading : Discover how to navigate effectively on the Le Bilan website and its sections
We regularly observe sites whose sitemap references several thousand URLs while half return a 404 code or a noindex status. The “Sitemaps” report in Google Search Console allows diagnosing these discrepancies: it shows the number of submitted URLs compared to the number of URLs actually indexed. A low ratio indicates a cleanliness issue with the file.
The best practice is to align the sitemap with the canonical tag strategy. If two URLs point to the same resource, only the canonical version appears in the file. This consistency reduces waste and speeds up the discovery of new or updated pages.
Recommended read : Understanding the Causes of a Hydrangea Turning Green and How to Remedy It
To see a concrete example of a well-structured sitemap, the sitemap page of Success Man illustrates this logic of filtered and prioritized URLs on a content site.
Lastmod and Changefreq Tags in a Sitemap: What Google Really Uses
The sitemap protocol specification includes several optional tags. Not all carry the same weight for search engines.
The lastmod tag influences crawl prioritization when it reflects an actual modification date. Google has confirmed that it uses this information to decide which URLs to revisit first. Conversely, an artificially updated lastmod date (for example, with every site deployment, without content change) eventually gets ignored.
The changefreq tag (daily, weekly, monthly) is largely disregarded by Google. We consider it noise in the file. The priority tag, intended to indicate the relative importance of a URL, suffers the same fate: Google does not take it into account for its crawl or ranking decisions.
In practice, an effective sitemap is limited to the following tags for each entry:
- The loc tag, which contains the absolute canonical URL of the page
- The lastmod tag, updated only when there is a substantial content change
- The hreflang tags (via xhtml:link) for multilingual sites, to signal alternative language versions directly in the sitemap
Specialized Sitemaps: Images, Videos, and News
The protocol is not limited to HTML pages. Extensions allow for declaring rich content that crawlers might not always discover through standard crawling.
An image sitemap declares the visuals associated with each URL. This improves coverage in Google Images, particularly useful for e-commerce sites or portfolios. Without this declaration, dynamically loaded images (lazy loading, JavaScript) may remain invisible to the bot.
The video sitemap works on the same principle: it provides metadata (title, description, duration, thumbnail URL) that the bot cannot extract from an embedded player. News sites have a dedicated format (news sitemap) that signals articles published in the last 48 hours, speeding up their appearance in Google News.
When to Deploy a Specialized Sitemap
An image or video sitemap is only justified if the multimedia content represents a real traffic lever. A text blog without ambitions on Google Images has no interest in maintaining this file. Each additional sitemap adds a maintenance burden: outdated URLs, expired metadata, non-updated files.
XML Sitemap and Internal Linking: Two Complementary, Not Substitutable Mechanisms
A common mistake is to treat the sitemap as a substitute for internal linking. The sitemap facilitates URL discovery, but internal linking distributes authority among pages. A bot that discovers a page via the sitemap without any internal links pointing to it will consider it orphaned, which weighs on its ranking potential.
The reverse scenario also poses a problem. A site with dense linking but no sitemap may see its deep pages discovered late, especially if the hierarchy exceeds three or four levels deep. Large sites or those whose structure heavily relies on JavaScript benefit the most from an up-to-date sitemap.
Diagnosing Orphaned Pages via the Sitemap
We recommend cross-referencing the URL list from the sitemap with a complete crawl of the site (using a tool like Screaming Frog or Sitebulb). URLs present in the sitemap but absent from the crawl are orphaned pages. Crawled URLs but absent from the sitemap are potentially forgotten for indexing.
- Pages in the sitemap but not found by crawl: add at least one contextual internal link
- Crawled pages but absent from the sitemap: check their indexing status and include them if they are canonical
- Pages in the sitemap with a noindex status or a redirect: remove them immediately from the file
This regular cross-referencing ensures that the sitemap remains a true reflection of the site’s actual structure, and not a technical artifact disconnected from the editorial reality.
A clean sitemap, limited to indexable URLs and enriched with reliable lastmod dates, remains one of the simplest technical levers to implement. Its effectiveness entirely depends on its maintenance: a file generated once and then forgotten loses all utility within a few months.