Sitemap Atomicity

The core principle of atomicity: the sitemap file is swapped in a single, indivisible operation. There is no intermediate state where a crawler could download a partially written file.

• Write to a temporary file first, never overwrite the live file in place
• Use filesystem rename operations which are atomic on POSIX-compliant systems
• Object storage platforms like S3 support atomic PUT operations with conditional headers
• Crawlers always receive either the complete old version or the complete new version

Without atomicity, a crawler downloading a sitemap mid-write receives a truncated XML document. This causes XML parsing errors and can lead to dropped URLs from the index.

• Truncated sitemaps fail XML schema validation
• Search engines may discard the entire file if malformed
• Corrupted reads waste crawl budget on unparseable resources
• Recovery requires waiting for the next crawl cycle, potentially days later

A classic Unix approach: generate the new sitemap at a temporary path, then atomically update a symbolic link to point to the new file. The webserver serves the file at the symlink target.

• ln -sf /tmp/sitemap-new.xml /var/www/sitemap.xml
• The symlink update is an atomic filesystem operation
• Rollback is trivial: point the symlink back to the previous version
• Works seamlessly with Nginx and Apache static file serving

Cloud object stores like Amazon S3 and Google Cloud Storage provide native atomicity for PUT operations on a single key. The new object becomes available only after the upload completes successfully.

• Use If-None-Match or If-Match conditional headers for optimistic concurrency
• S3 guarantees read-after-write consistency for new objects in all regions
• Combine with CDN cache invalidation to propagate the update globally
• No partial reads: the object is invisible until fully committed

Atomicity implies a rollback guarantee: if generation fails at any point, the live sitemap remains untouched. This eliminates the risk of serving an empty or broken file due to a pipeline error.

• Generation failure leaves the previous valid sitemap in place
• No need for emergency manual intervention after pipeline crashes
• Idempotent retries can safely re-attempt generation
• Maintains crawler trust by never serving invalid XML

Atomic origin updates must be paired with coordinated CDN purging to prevent edge nodes from serving stale or mixed versions. A two-phase approach ensures global consistency.

• Phase 1: Perform the atomic swap at origin
• Phase 2: Issue a targeted cache invalidation for the sitemap path
• Use surrogate keys for precise purging without flushing the entire cache
• Verify with curl -H "Pragma: no-cache" against edge nodes post-deployment