Readiness Probe

A readiness probe's primary function is to act as a traffic gatekeeper. When a probe succeeds, the container's pod is added as an endpoint to the associated Kubernetes Service. If it fails, the pod is removed from service endpoints, preventing traffic from being sent to an unprepared or malfunctioning container. This is distinct from a liveness probe, which determines if a container should be restarted.

Kubernetes supports three primary probe mechanisms, defined in a container's spec:

• HTTP GET Probe: Sends an HTTP request to a specified path and port; succeeds on a 2xx or 3xx status code.
• TCP Socket Probe: Attempts to open a TCP connection to a specified port; succeeds if the connection is established.
• Exec Probe: Executes a specified command inside the container; succeeds if the command exits with status code 0.

Key configuration parameters include initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold, and failureThreshold.

Readiness probes are essential for safe rolling updates and canary deployments. During a rolling update, new pods are created but only receive traffic once their readiness probe passes. If a new pod's probe fails, the update pauses, preventing a cascading failure. This allows for progressive delivery by ensuring only healthy pods serve user requests, enabling zero-downtime deployments.

Probes are crucial for stateful applications like databases, caches, and LLM inference servers that require initialization. For example, a vector database pod may need to load a large index into memory before it can serve queries. A correctly configured readiness probe with a sufficient initialDelaySeconds prevents the Service from routing traffic until this warm-up is complete, avoiding timeouts and errors for end-users.

Misconfiguration leads to operational issues:

• Shallow Checks: Probing a trivial endpoint (e.g., /) that doesn't verify dependent services (e.g., model loaded, database connection).
• Insufficient Initial Delay: Causing premature failure for services with long startup times.
• Overlapping with Liveness: Using the same check for both readiness and liveness can cause unnecessary restarts under load.
• Resource-Intensive Checks: An exec probe running a heavy script can exacerbate resource pressure on a struggling pod.

Readiness probes operate within a broader ecosystem of traffic and deployment controls:

• Liveness Probe: Determines if a container should be restarted (survival).
• Health Check: A generic term for any endpoint or check used to assess application status.
• Load Balancer: Relies on readiness probe status to update its pool of healthy backends.
• Service Mesh: Often provides more advanced health checking and traffic shifting capabilities on top of basic Kubernetes probes.

A readiness probe is a periodic check executed by the kubelet (the node agent) against a container. Its primary purpose is to signal when a pod is ready to start serving traffic. Unlike a liveness probe (which determines if a container should be restarted), a failed readiness probe does not restart the container; it simply removes the pod's IP address from the endpoints of all matching Services. This prevents load balancers from routing user requests to pods that are still booting, loading a large model, or warming up caches, ensuring users only hit healthy, responsive endpoints.

• Probe Types: HTTP GET, TCP Socket, or Exec command.
• Critical for LLMs: Large models can take minutes to load into GPU memory; a readiness probe gates traffic until this is complete.

Readiness probes are defined in a pod's container specification with several key parameters that control their behavior:

• initialDelaySeconds: The number of seconds to wait after the container starts before initiating probes. Crucial for LLMs to allow for model loading.
• periodSeconds: How often (in seconds) to perform the probe.
• timeoutSeconds: Number of seconds after which the probe times out.
• successThreshold: Minimum consecutive successes for the probe to be considered successful after having failed.
• failureThreshold: Number of consecutive failures required for the probe to be considered failed.

Example Configuration for an LLM API:

yamlCopy
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8000
  initialDelaySeconds: 180  # Allow 3 minutes for model load
  periodSeconds: 10
  failureThreshold: 3

The readiness probe is a foundational component for advanced traffic and deployment strategies. Its status directly interfaces with Kubernetes Services and Ingress controllers.

• Service Endpoints: A Kubernetes Service continuously watches for pods matching its selector. It only adds pods with passing readiness probes to its active endpoints list.
• Rolling Updates & Canary Deployments: During a rolling update, new pods are created. They remain isolated from live traffic until their readiness probes pass, ensuring a smooth transition. For canary deployments, readiness probes validate the new version's health before any traffic is shifted via traffic splitting.
• Load Balancers: Cloud load balancers (e.g., AWS ALB, GCP Cloud Load Balancing) that integrate with Kubernetes use endpoint readiness to determine where to send requests, preventing 503 errors and connection drops.

A well-designed readiness check for an LLM service must verify more than just a running process. It should assert the application is functionally ready.

Best Practices:

• Endpoint Logic: The /health/ready endpoint should check critical dependencies: model loaded in memory, vector database connection established, GPU availability, and any warm-up caches (e.g., KV caches for inference optimization).
• Lightweight & Fast: The check must be computationally cheap and return quickly (e.g., < 1 second) to avoid becoming a bottleneck.
• Distinct from Liveness: Use a separate /health/live endpoint for liveness that checks if the process is running, while readiness performs deeper checks.
• Graceful Shutdown: During termination, the kubelet stops sending traffic as soon as the pod is marked for deletion, but the probe should also be designed to fail quickly if the service is draining connections.

Understanding why a readiness probe fails is key to maintaining high availability.

Typical Failure Causes:

• Insufficient initialDelaySeconds: The most common issue. The probe starts before the LLM finishes loading its weights.
• Resource Contention: The pod may be CPU throttled or waiting for a GPU, causing health check timeouts.
• External Dependency Failure: The probe checks a downstream service (e.g., a tokenizer service, feature store) that is unavailable.
• Buggy Probe Logic: The health check itself has an error or infinite loop.

Debugging Commands:

• kubectl describe pod <pod-name>: View probe status, last error, and events.
• kubectl logs <pod-name>: Check application logs for errors during startup.
• kubectl exec -it <pod-name> -- curl localhost:8000/health/ready: Manually test the probe endpoint from within the pod.

Readiness probes do not operate in isolation; they are part of a broader ecosystem of cloud-native resilience patterns.

• Liveness Probe: Determines if a container needs to be restarted. Used for catching deadlocks or stalled processes where the app is running but not functional.
• Startup Probe: Used for legacy applications that require extra long startup times. Disables liveness and readiness checks until it succeeds once.
• Horizontal Pod Autoscaler (HPA): Scales the number of pod replicas based on metrics. New pods created by the HPA must pass their readiness probes before receiving traffic.
• Service Mesh (e.g., Istio): Often adds its own layer of health checking and outlier detection, which can work in concert with or independently of Kubernetes readiness probes.
• Circuit Breaker: A pattern implemented at the service mesh or application level to stop sending requests to a failing pod, complementing the probe's traffic removal function.

A Kubernetes mechanism that determines if a container is running. If this probe fails, the kubelet kills and restarts the container. It addresses process crashes or deadlocks, distinct from a readiness probe which checks for operational initialization.

• Purpose: Detect and recover from unrecoverable application states.
• Action on Failure: Container restart.
• Common Checks: Simple endpoint response, process status.

A general term for any periodic test performed by an orchestrator or load balancer to verify an application instance's operational status. In Kubernetes, this is implemented via liveness and readiness probes. More broadly, health checks are foundational to high-availability architectures.

• Scope: Broader than Kubernetes probes; used in cloud load balancers (AWS ELB, GCP Cloud Load Balancing).
• Function: Determines if a host should receive traffic or be marked unhealthy.

A Kubernetes controller that automatically scales the number of pods in a deployment or replica set based on observed CPU utilization, memory consumption, or custom metrics. It works in concert with readiness probes; pods must be ready before they can be considered part of the scalable pool and receive traffic.

• Scaling Trigger: Metrics exceeding defined thresholds.
• Integration: Relies on the Metrics Server or custom metrics APIs.
• Goal: Maintain application performance under variable load.

The default Kubernetes deployment strategy where new versions of an application are gradually rolled out by incrementally replacing old pods with new ones. Readiness probes are critical here; the rollout pauses if a new pod fails its readiness check, preventing a faulty version from receiving user traffic.

• Mechanism: Controlled pod termination and creation.
• Benefit: Zero-downtime deployments and easy rollback.
• Dependency: Requires correctly configured readiness probes for stability.

A dedicated infrastructure layer (e.g., Istio, Linkerd) for managing service-to-service communication in a microservices architecture. It often implements advanced traffic management and observability, complementing Kubernetes' built-in probes with more sophisticated health checking, latency-aware load balancing, and circuit breaking.

• Enhanced Probes: Can perform protocol-specific health checks (gRPC, HTTP/2).
• Traffic Control: Enables fine-grained canary deployments and fault injection.

A design pattern for preventing cascading failures in distributed systems. When a downstream service fails repeatedly, the circuit breaker "opens" and fails fast, avoiding overwhelming the struggling service. While readiness probes remove unhealthy pods, circuit breakers protect clients from repeatedly calling unhealthy endpoints.

• States: Closed, Open, Half-Open.
• Implementation: Found in service meshes, API gateways, and client libraries (e.g., Resilience4j).
• Synergy: Works with probes to create resilient service communication.