Graceful Shutdown

In Kubernetes, a graceful shutdown is initiated when a pod is terminated. The sequence is:

• The kubelet sends a SIGTERM signal to the main process (PID 1) in each container.
• A terminationGracePeriodSeconds (default 30s) countdown begins.
• The application should complete in-flight requests, close network listeners, and release resources.
• If the process is still running after the grace period, SIGKILL is sent for forced termination.
• The PreStop lifecycle hook can be used to execute a custom command or HTTP request before SIGTERM, allowing for complex cleanup sequences.

Docker and other OCI-compliant runtimes (containerd, CRI-O) manage graceful shutdown at the container level.

• The docker stop command sends SIGTERM, waits for a configurable timeout (default 10s), then sends SIGKILL.
• The timeout can be set via the --time flag or the STOP_TIMEOUT instruction in a Dockerfile.
• The application inside the container must handle SIGTERM. Best practice is to use a process manager like tini (init=true) as the entrypoint to properly forward signals to child processes.
• Container orchestrators like Kubernetes interact with these runtime APIs to manage pod termination.

Major cloud platforms provide metadata services to notify instances of impending termination, allowing for graceful shutdown before hardware reclamation.

• AWS EC2: Spot Instances and Auto Scaling groups send a termination notice via the Instance Metadata Service (IMDS) at http://169.254.169.254/latest/meta-data/spot/instance-action. Applications can poll this endpoint and have 120 seconds (typically) to shut down.
• Google Cloud: Preemptible VMs receive an ACPI G3 Soft Off signal 30 seconds before termination.
• Azure: Spot VMs receive a notification via the Azure Metadata Service and have 30 seconds to complete cleanup.
• This pattern is essential for saving state, draining queues, and deregistering from load balancers.

Application frameworks implement graceful shutdown within their request/response cycle.

• Node.js (Express/Fastify): The server .close() method stops accepting new connections but keeps existing connections open until they complete. Signal handlers for SIGTERM/SIGINT trigger this close.
• Python (ASGI/Uvicorn): ASGI servers like Uvicorn have a shutdown_timeout config. On signal, they stop accepting connections, wait for ongoing requests, then terminate workers.
• Java Spring Boot: Actuator's /actuator/shutdown endpoint (if enabled) or a DisposableBean interface allows custom cleanup. The embedded Tomcat/Jetty server will stop gracefully on a JVM shutdown hook.
• Go (http.Server): The Shutdown(context.Context) method performs a graceful shutdown, using a context to set a deadline.

Graceful shutdown for queue consumers is vital to prevent message loss or reprocessing.

• Apache Kafka: Consumers use a Consumer.close() which triggers a final offset commit and leaves the group gracefully.
• RabbitMQ: Consumers should acknowledge (ACK) outstanding messages and close the channel before disconnecting. Missed ACKs will cause the broker to requeue messages.
• AWS SQS: Long-polling workers should finish processing the current batch of messages and delete them from the queue before terminating.
• The pattern involves: 1. Stopping the message fetch loop, 2. Completing in-flight message processing, 3. Performing final state commits (e.g., offsets), 4. Closing the connection.

Service meshes add a proxy sidecar (e.g., Envoy) to each pod, which must also shut down gracefully to avoid interrupting traffic.

• During pod termination, the kubelet sends SIGTERM to both the main application container and the sidecar.
• The sidecar proxy must stop accepting new connections but continue to allow established connections to complete (connection draining).
• Istio's pilot-agent sets a drainDuration in the Envoy configuration. The application's PreStop hook may need to wait for the proxy to drain (e.g., sleep 20).
• This ensures the proxy doesn't cut off traffic to the application mid-request during its own cleanup phase.

Kubernetes mechanisms that allow code execution at specific points in a container's lifecycle. The PreStop hook is the primary enabler of a graceful shutdown, as it is a blocking call executed before the container termination signal (SIGTERM).

• PostStart Hook: Executes immediately after a container is created.
• Use Case: A PreStop hook can run a custom script to drain connections, flush logs to disk, or notify dependent services of impending termination, ensuring a clean handoff.

A Kubernetes API object that limits the number of concurrent voluntary disruptions to pods in an application. It works in tandem with graceful shutdown to ensure high availability during cluster operations like node maintenance or rolling updates.

• Voluntary Disruptions: Actions initiated by the cluster operator (e.g., draining a node).
• Key Parameters: minAvailable or maxUnavailable define how many pods can be down simultaneously.
• Interaction: The orchestrator respects the PDB, ensuring pods are terminated gracefully one by one without violating the availability guarantee.

Health checks that determine a pod's operational status. They are essential for orchestrating traffic during a rolling update, which relies on graceful shutdowns.

• Readiness Probe: Signals when a pod is ready to receive traffic. During an update, new pods must pass readiness checks before being added to the service load balancer.
• Liveness Probe: Determines if a pod is healthy. If it fails, the pod is killed and restarted.
• Graceful Shutdown Context: As old pods are terminated gracefully, the service mesh or kube-proxy uses failing readiness probes to stop sending them new requests, allowing in-flight work to complete.

A resilience pattern for microservices that prevents a failing or slow downstream service from causing cascading failures. It complements graceful shutdown in distributed systems.

• States: Closed (normal operation), Open (requests fail fast), Half-Open (testing for recovery).
• Synergy with Shutdown: When a service instance begins its graceful shutdown, a circuit breaker in upstream services can quickly trip to 'Open' for that instance, redirecting traffic to healthy pods and preventing requests from being sent to a terminating endpoint.

The standard Kubernetes deployment strategy where pod instances are incrementally replaced with new versions. Graceful shutdown is the critical mechanism that makes this a zero-downtime operation.

• Process: The controller creates new pods, waits for them to become ready, then terminates old pods.
• Termination Grace Period: Each pod is given a terminationGracePeriodSeconds (default 30s) to complete its PreStop hook and shut down before being forcibly killed (SIGKILL).
• Outcome: Properly implemented graceful shutdowns ensure user sessions are not dropped and data integrity is maintained during the update.

A dedicated infrastructure layer that handles service-to-service communication. It provides advanced traffic management features that rely on and enhance graceful shutdown procedures.

• Connection Draining: Meshes often implement intelligent draining, holding terminating pods in a 'draining' state until all active connections are closed.
• Traffic Shifting: During a canary deployment, the mesh can precisely shift traffic away from pods slated for termination before the shutdown signal is sent.
• Observability: Provides detailed metrics on active connections and request completion during shutdown, verifying graceful behavior.