Readiness Probe

The primary function of a readiness probe is to control membership in a Service's endpoint list. A container is not added to load balancer rotation until the probe succeeds. If a running container's probe fails, it is immediately removed from the pool, preventing requests from being routed to a degraded instance. This is distinct from a liveness probe, which restarts a failed container; a readiness probe simply stops sending it traffic, allowing it to recover independently.

For machine learning serving, the most critical use case is blocking traffic until model weights are fully loaded into GPU or CPU memory. A large language model can take minutes to initialize. The readiness probe must check that:

• The model file is loaded and deserialized
• The inference engine (e.g., Triton, vLLM) has completed warm-up
• The first dummy inference request succeeds Without this, the container will accept requests and return errors or time out.

Three timing parameters govern probe behavior and must be tuned to the model's loading profile:

• initialDelaySeconds: The grace period before the first probe runs. Set this to the minimum expected model load time to avoid premature failure.
• periodSeconds: How often the probe executes. For static readiness, a longer interval (e.g., 10s) is acceptable.
• failureThreshold: The number of consecutive failures before the container is marked Unready. A value of 3 is common to tolerate transient blips.

Kubernetes supports three mechanisms to execute a readiness check:

• exec: Runs a command inside the container. Ideal for a Python script that calls the model's /health endpoint or checks for a marker file written after loading.
• httpGet: Performs an HTTP GET request. The model server must expose a lightweight /v1/health/ready endpoint that returns a 200 OK only when ready.
• tcpSocket: Checks if a TCP port is open. This is a coarse check that only verifies the process is listening, not that the model is loaded. Avoid for ML serving.

For models with extremely long initialization times, combine a readiness probe with a startup probe. The startup probe is only active during initial container startup and can have a longer failureThreshold and periodSeconds. The readiness probe is disabled until the startup probe succeeds. This prevents the container from being killed by the liveness probe during a legitimate, slow model-loading phase while still protecting against hung processes.

When a pod receives a SIGTERM during a rolling update, the readiness probe must fail immediately to drain traffic before the container terminates. The sequence is:

1. Kubernetes marks the pod as Terminating
2. The readiness probe fails, removing the pod from Service endpoints
3. The pod waits for terminationGracePeriodSeconds for in-flight requests to complete
4. The container is forcefully killed A preStop hook can explicitly fail the readiness endpoint to accelerate this drain.

Fine-tuning probe behavior is essential for model serving reliability. Key parameters include:

• initialDelaySeconds: Time to wait before starting probes, allowing the container process to begin
• periodSeconds: Frequency of probe execution (default: 10)
• timeoutSeconds: Maximum time to wait for a probe response before counting it as a failure
• failureThreshold: Number of consecutive failures before the probe is considered failed (for readiness) or the container is restarted (for liveness)
• successThreshold: Number of consecutive successes required after a failure to mark the container as ready again Misconfiguring these values can cause flapping—where containers oscillate between ready and not-ready states under load.

In service mesh architectures like Istio or Linkerd, readiness probes integrate with the sidecar proxy to control traffic routing at a finer granularity. The mesh's control plane watches Kubernetes readiness states and dynamically updates its routing tables. When a readiness probe fails:

• The endpoint is removed from the mesh's load balancing pool
• Existing connections may be drained gracefully via connection draining settings
• Circuit breakers can be triggered if failure rates exceed thresholds This prevents the mesh from routing inference requests to a model instance that is still loading weights or experiencing GPU memory pressure.

For model serving frameworks like NVIDIA Triton Inference Server or vLLM, readiness endpoints should validate more than just process health. A robust readiness check for ML workloads should verify:

• The model has been fully loaded into GPU memory and is ready for inference
• The KV cache is initialized and available
• Backend dependencies like feature stores or vector databases are reachable
• GPU drivers and CUDA libraries are accessible A common pattern is exposing a /v1/health/ready endpoint that returns HTTP 200 only when the model can successfully process a lightweight inference request, confirming end-to-end readiness.

A Pod Disruption Budget (PDB) works alongside readiness probes to protect model serving availability during voluntary disruptions like node drains or cluster upgrades. A PDB specifies the minimum number of replicas that must remain available, defined by minAvailable or maxUnavailable. When a node is drained, the eviction API respects the PDB and only evicts pods if the budget is maintained. Combined with proper readiness probes, this ensures that traffic is never routed to terminating instances and that enough healthy replicas remain to handle the inference load during maintenance operations.