Sitemap Sharding

The partition key is the logical rule used to distribute URLs across shards. Common strategies include:

• Content Type: Separate sitemaps for products, articles, and categories.
• ID Range: Numeric ranges (e.g., products-1-50000.xml, products-50001-100000.xml).
• Date-Based: Partitioning by publication date for news archives.
• Alphabetical: Grouping by the first letter of a URL slug. The key must ensure even distribution to prevent some shards from hitting the 50,000 URL limit while others remain underutilized.

Sharding requires a sitemap index file to act as the single entry point for search engines. The index lists every shard file and its <lastmod> timestamp.

• Crawlers discover the index via robots.txt or direct submission.
• The index itself is subject to the same 50MB uncompressed size limit.
• A single index can reference up to 50,000 sitemap shards, theoretically supporting 2.5 billion URLs.

For sites with frequent content changes, shards are generated on-demand or incrementally rather than as static files.

• Database-to-Sitemap Pipelines query the backend using the partition key to populate each shard.
• Delta Sitemaps can be combined with sharding: only modified shards are regenerated.
• Edge caching with short TTLs ensures crawlers receive fresh data without overwhelming the origin server.

Sharding directly improves crawl budget allocation by providing granular signals to search engines:

• High-priority shards (e.g., new products) can have more recent <lastmod> dates.
• Low-value shards (e.g., archived content) can be updated less frequently.
• Search engines may prioritize crawling specific shards based on their historical change frequency.
• Isolating problematic URL patterns into separate shards prevents them from contaminating the crawl signals of high-value content.

Sitemap atomicity ensures a crawler never sees a partially updated shard. Best practices include:

• Writing the complete shard to a temporary file first, then atomically renaming it.
• Using object storage with conditional writes to prevent concurrent modification.
• Implementing health checks that validate XML schema compliance before publishing.
• Coordinating shard updates with cache invalidation to prevent stale data from being served.

Instrumenting shard generation pipelines is critical for debugging at scale:

• Generation latency per shard to identify slow database queries.
• Error rates by partition key to detect data anomalies.
• Submission status tracking to confirm search engines successfully ingested each shard.
• Log file analysis to verify that crawlers are discovering and fetching all shards proportionally.

The parent XML file that orchestrates sharded sitemaps. A sitemap index lists up to 50,000 individual sitemap files, enabling a single domain to submit 2.5 billion URLs to search engines. Without an index, sharded files remain undiscoverable.

• References child sitemaps via <sitemap> elements
• Each <sitemap> contains a <loc> and optional <lastmod>
• Must reside at the domain root or be submitted via robots.txt

The hard constraint that necessitates sharding. Each uncompressed sitemap file must contain no more than 50,000 URLs and must not exceed 50MB when uncompressed. Exceeding either limit causes search engines to reject the file entirely.

• Both limits apply simultaneously; hitting either triggers rejection
• Compression via Gzip reduces bandwidth but the 50MB limit applies to the uncompressed payload
• Sharding by ID range (e.g., 0–49,999, 50,000–99,999) is the most common partitioning strategy

A sharding optimization that contains only URLs added, modified, or deleted since the last full generation. Instead of regenerating all shards, delta sitemaps allow crawlers to process incremental changes, dramatically reducing crawl latency for frequently updated sections.

• Pairs with a full baseline sitemap set
• Uses <lastmod> timestamps to signal freshness
• Ideal for news sites, e-commerce inventory, and job boards

The approximate number of URLs a search engine will crawl on a site within a given timeframe. Sharding directly impacts crawl budget allocation: high-priority shards (e.g., product pages) should be listed first in the sitemap index to signal importance.

• Crawl budget is a function of site authority and server health
• Wasted budget on low-value or duplicate URLs reduces indexation of critical pages
• Sharding by content type enables granular budget steering

The application of Gzip encoding to reduce sitemap file size before transmission. For sharded architectures with thousands of files, compression mitigates bandwidth costs and accelerates delivery to crawlers. Search engines accept .xml.gz files natively.

• Typically achieves 80–90% size reduction
• Must be decompressed by the crawler before parsing
• Combine with CDN edge caching for optimal delivery

The process of standardizing URLs to a consistent canonical form before assigning them to shards. Without normalization, the same resource might appear in multiple shards under different representations, wasting crawl budget and diluting ranking signals.

• Lowercase scheme and host
• Remove default ports (:80, :443)
• Decode safe characters and strip trailing slashes consistently
• Essential for deterministic shard key assignment