Sitemap Cache-Control

The max-age directive specifies the maximum time in seconds a resource is considered fresh. For sitemaps, a value like 3600 (1 hour) is common for dynamic sites, while 86400 (24 hours) suits static sites. Browsers and intermediate caches honor this before revalidating with the origin server.

• Syntax: Cache-Control: max-age=3600
• Freshness Window: Defines the period during which a cached copy is served without contacting the origin.
• Trade-off: Shorter values increase server load but ensure crawlers see recent URL changes.

Content Delivery Networks cache sitemaps at edge nodes globally, reducing origin server load and latency for crawlers. A stale-while-revalidate directive allows serving a stale cached copy while asynchronously fetching a fresh version in the background.

• Edge TTL: Set via s-maxage to control CDN-specific caching independently of browser cache.
• Cache Key: Normalize the cache key to ignore irrelevant query parameters or vary on Accept-Encoding.
• Purge API: Use programmatic cache invalidation when content updates trigger sitemap regeneration.

The ETag (Entity Tag) header provides a unique identifier for a specific version of a sitemap. Crawlers send an If-None-Match request header with the stored ETag; the server responds with 304 Not Modified if the sitemap hasn't changed, saving bandwidth.

• Validation: Enables efficient revalidation without retransmitting the full file.
• Generation: Often derived from a hash of the sitemap content or a version ID.
• Last-Modified: A fallback mechanism using timestamps, though less precise than ETags for sub-second changes.

When a sitemap exceeds its max-age, it becomes stale. The stale-if-error directive allows caches to serve a stale sitemap if the origin server returns a 5xx error, ensuring crawlers still receive a valid sitemap during outages.

• Grace Period: stale-if-error=86400 permits serving a day-old sitemap during server failures.
• must-revalidate: Forces caches to check with the origin before using a stale copy, ensuring strict freshness.
• proxy-revalidate: Applies revalidation requirements specifically to shared caches like CDNs.

Googlebot and Bingbot respect standard HTTP caching headers but may apply their own heuristics. Google typically recrawls sitemaps based on the lastmod date and observed update frequency, not solely on cache headers.

• Crawl Frequency: Sitemaps with frequent lastmod changes are recrawled more often, regardless of cache settings.
• Sitemap Polling: Search engines poll sitemap URLs periodically; cache headers influence the efficiency of these polls.
• IndexNow Protocol: Bypasses caching entirely by pushing URL updates directly to search engines via API.

Programmatic cache invalidation ensures crawlers receive updated sitemaps immediately after content changes. Common patterns include purge-by-tag, purge-by-URL, and time-to-live (TTL) expiration.

• Event-Driven Purge: Trigger cache invalidation via webhook when a content management system publishes or updates content.
• Surrogate Keys: Assign tags like sitemap-products to purge all related cached sitemap fragments simultaneously.
• Versioned URLs: Append a version hash to sitemap URLs (e.g., sitemap.xml?v=abc123) to force cache misses on update.