Immutable Infrastructure

The cardinal rule of immutability: never patch, update, or SSH into a running instance. Any change, from a security patch to a configuration tweak, triggers the creation of a new, versioned artifact from a golden image. The old instance is terminated and replaced wholesale. This eliminates configuration drift—the slow, undocumented divergence of servers from their intended state that causes the majority of production incidents. In a disconnected Kubernetes environment, this means a new container image is built, pushed to a private registry, and rolled out via a declarative deployment.

All instances are derived from a single, version-controlled golden image that acts as the cryptographic source of truth. This image is built once via an automated pipeline, undergoes security scanning, and is then promoted through environments. Key properties:

• Versioned and signed: Every image has a digest (e.g., sha256:...) for cryptographic verification
• Pre-baked dependencies: All libraries, drivers (e.g., NVIDIA GPU Operator components), and configurations are baked in at build time, not installed at runtime
• No runtime agents: Eliminates the need for configuration management tools like Ansible or Puppet to converge state on live nodes

The desired state of infrastructure is defined declaratively in code (YAML manifests, Terraform, Helm charts) and stored in a Git repository. A reconciliation loop—such as a GitOps operator like ArgoCD—continuously compares the declared state against the actual cluster state. Any manual change to a running pod is automatically reverted. This pattern is critical for air-gapped AI clusters where static manifest bundling ensures the entire desired state is self-contained and deployable without external API calls.

Immutable instances are treated as cattle, not pets—they are disposable and carry no unique identity. To achieve this, persistent state must be strictly separated from compute:

• Stateful data (model weights, vector databases, user uploads) is stored on external, persistent volumes like Local Persistent Volumes or network-attached storage
• Stateless compute (inference servers like vLLM or Triton Inference Server) runs on ephemeral pods that can be destroyed and recreated without data loss
• Bootstrapping at startup pulls necessary configuration from a secure secret store (e.g., Sealed Secrets), not from local disk

Since instances cannot be accessed for debugging, telemetry must be externalized from the moment of creation. An immutable infrastructure mandates:

• Structured logs streamed to a centralized, append-only log store (e.g., Loki or Elasticsearch) outside the ephemeral instance
• Metrics scraped by Prometheus from a known, pre-configured endpoint; no agent installation post-boot
• Traces emitted via OpenTelemetry to reconstruct request flows across replaced instances
• This ensures that when an instance is terminated, its forensic record remains intact for root cause analysis

Updates are performed via a rolling replacement strategy, not an in-place upgrade. In Kubernetes, this is natively supported through Deployments and StatefulSets:

• A new ReplicaSet is created with the updated image digest
• New pods are spun up and pass readiness probes before receiving traffic
• Old pods are gracefully terminated only after new ones are healthy
• If the new version fails health checks, the rollout is automatically halted, and the previous immutable version continues serving—providing an instant, zero-downtime rollback mechanism

A release strategy where two identical environments—blue (current) and green (new)—run in parallel. Traffic is switched atomically to the green environment after validation.

• Eliminates downtime during updates.
• Instant rollback by reverting traffic to the blue stack.
• Relies entirely on immutable instances being replaced, not patched.
• Essential for zero-downtime model serving in air-gapped clusters.

A discipline where servers are routinely terminated and rebuilt from scratch, never patched in place. The name derives from the mythical phoenix rising from ashes.

• Enforces strict immutability by destroying drift.
• Validates that the Golden Image Pipeline is always functional.
• Limits the maximum lifespan of any single instance.
• Prevents long-lived, unmanaged 'snowflake' servers.

The practice of managing all system configuration in version-controlled, declarative files. This separates ephemeral compute from persistent state.

• Terraform manages the underlying infrastructure.
• Helm or Kustomize defines the Kubernetes application state.
• Changes are applied via a GitOps reconciliation loop.
• No manual kubectl edit or SSH ad-hoc fixes are permitted.

A design philosophy treating servers and containers as cattle, not pets. Any instance can be terminated and replaced without loss of critical state.

• State is externalized to persistent volumes or databases.
• Horizontal Pod Autoscaler (HPA) manages ephemeral replicas.
• Node failures trigger automatic replacement via Karpenter.
• Critical for maintaining operational security in sovereign AI deployments.