Graceful Shutdown

The primary mechanism of graceful shutdown where the server stops accepting new connections while continuing to process all requests already in its queue. The load balancer must receive a SIGTERM signal and immediately remove the endpoint from its healthy pool. The server then enters a drain phase, waiting for all active inference calls—including long-running generative completions—to finish before exiting. Kubernetes provides a terminationGracePeriodSeconds field to define the maximum wait time before a forced SIGKILL.

A clean termination sequence that releases GPU memory, database connections, and file handles without corruption. The server must:

• Unload model weights from VRAM to free GPU memory for other pods
• Close connection pools to vector databases and feature stores gracefully, sending proper FIN packets
• Flush write buffers to ensure all telemetry and prediction logs are persisted
• Deregister from service discovery (Consul, etcd) to prevent routing to a dead instance Failure to deallocate can cause GPU memory leaks and orphaned connections.

A Kubernetes lifecycle hook that executes a command or HTTP call before the container receives SIGTERM. This is the ideal place to trigger custom cleanup logic:

• Sleep for 5-10 seconds to allow the kube-proxy to update iptables rules and stop sending traffic
• Call a /drain endpoint on the model server to initiate graceful shutdown programmatically
• Write a checkpoint of any in-memory state to persistent storage Without a PreStop hook, the pod may receive traffic for several seconds after SIGTERM, causing dropped requests.

During shutdown, the server must flip its readiness probe to return a non-200 status code. This signals the kubelet to remove the pod from the Service endpoint list. The liveness probe should remain healthy until all requests are drained to prevent Kubernetes from killing the container prematurely. A common pattern is to have a /healthz endpoint return 503 Service Unavailable during the drain phase while /livez continues returning 200 OK until the process exits.

For gRPC and HTTP/2 connections using multiplexed streams, graceful shutdown requires sending a GOAWAY frame to the client. This frame informs the client that the server will not accept new streams on the existing connection but will finish processing all open streams. The client then transparently re-establishes a new connection to a different backend. Without GOAWAY, persistent Keep-Alive connections can continue routing traffic to a terminating instance long after the drain phase begins.

The terminationGracePeriodSeconds must exceed the P99 prediction latency of the model. If a generative model has a worst-case completion time of 30 seconds, a grace period of 60 seconds provides a safety margin. The calculation is:

• Grace Period = P99 Latency + PreStop Sleep + Buffer
• A buffer of 10-15 seconds accounts for network jitter and slow clients If the grace period is too short, long-running predictions are killed mid-inference, wasting GPU compute and returning partial results to the client.

The POSIX signal sent by container orchestrators to initiate a controlled shutdown. A well-behaved model server must trap SIGTERM and execute a shutdown sequence:

• Stop accepting new inference requests on the gRPC or HTTP port
• Complete all in-flight prediction requests within the termination grace period
• Flush metrics and close tracing spans
• Release GPU memory and close database connections
• Exit with code 0 Failure to handle SIGTERM results in SIGKILL after the grace period, forcibly dropping requests.

The process of allowing existing TCP connections to complete before a server instance is terminated. In a load-balanced model serving setup, connection draining works in concert with health checks:

1. The load balancer stops forwarding new connections to the draining instance
2. Existing long-lived gRPC streams or HTTP keep-alive connections are allowed to finish their current request
3. After a configurable timeout, remaining connections are forcefully closed This pattern is essential for stateful streaming inference where a single connection may handle multiple sequential predictions.

A Kubernetes lifecycle hook that executes a command or HTTP call before the container receives SIGTERM. Common PreStop hook patterns for model servers:

• sleep 15 to buy time for the kube-proxy to update iptables rules and stop routing traffic
• A custom script that calls an internal admin endpoint to set the instance to draining mode
• Flushing in-memory feature cache to a persistent store PreStop hooks run to completion before the termination grace period clock starts, providing a critical buffer for traffic to drain.

The time window between SIGTERM and forced SIGKILL, configured via terminationGracePeriodSeconds in a Pod spec. This value must be tuned to your model's inference latency profile:

• For real-time models with P99 latency of 50ms, a 30-second grace period is generous
• For large language models generating long sequences, the grace period must exceed the maximum token generation time
• Include buffer time for KV cache serialization if you're checkpointing state If in-flight requests consistently exceed the grace period, you'll observe SIGKILL-induced errors in client-side telemetry.