Readiness Probe

A readiness probe's primary function is to act as a traffic gatekeeper for a Kubernetes Service. When a pod's readiness probe succeeds, the pod's IP address is added to the Service's endpoints list, making it eligible to receive traffic from the Service's load balancer. If the probe fails, the pod's IP is removed, effectively taking it out of rotation. This prevents users from hitting a pod that is still loading dependencies, warming caches, or establishing database connections.

• Key Distinction: Unlike a liveness probe (which determines if a container should be restarted), a readiness probe determines if it should receive traffic.
• Use Case: A web server pod might need 30 seconds to load a large configuration file and connect to a database. Its readiness probe should only succeed after these initialization steps are complete.

Kubernetes supports three primary mechanisms for implementing a readiness check, each defined in the pod's specification under spec.containers[].readinessProbe.

• HTTP GET Probe: The most common type. Kubernetes sends an HTTP GET request to a specified path and port on the container. A response with an HTTP status code between 200 and 399 is considered a success. Ideal for HTTP/HTTPS services.
• TCP Socket Probe: Kubernetes attempts to open a TCP connection to a specified port on the container. Success is defined by establishing a connection. Used for non-HTTP services like databases (e.g., PostgreSQL, Redis).
• Exec Probe: Kubernetes executes a specified command inside the container. A zero exit code indicates success. This offers maximum flexibility for custom health logic (e.g., checking for a specific file, running a database query script).

Each probe type allows configuration of critical parameters: initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold, and failureThreshold.

The behavior of a readiness probe is finely controlled by a set of timing and threshold parameters, which are essential for robust configuration.

• initialDelaySeconds: The number of seconds to wait after the container starts before initiating the first probe. Crucial for allowing slow-starting applications to initialize.
• periodSeconds: How often (in seconds) to perform the probe. Default is 10 seconds.
• timeoutSeconds: Number of seconds after which the probe times out. Default is 1 second.
• successThreshold: Minimum consecutive successes for the probe to be considered successful after having failed. Default is 1.
• failureThreshold: When a probe fails, Kubernetes will retry this many times before marking the pod as Unready. Default is 3.

Example Scenario: A Java application with a 45-second JVM startup might use initialDelaySeconds: 50, periodSeconds: 5, failureThreshold: 3. This gives it time to start, checks frequently, and allows for brief transient failures.

The readiness probe is an integral part of the pod's lifecycle and interacts with other Kubernetes controllers and objects.

• Service Endpoints Controller: Continuously evaluates pod readiness to update the Service's Endpoints object.
• Rolling Updates: During a deployment update, new pods are brought online. They must pass their readiness probe before being added to the Service, and old pods are removed only after the new ones are ready. This ensures zero-downtime deployments.
• Pod Status: A pod with a failing readiness probe has its Ready condition set to False. This status is visible via kubectl get pods.
• Startup Probes: For legacy applications with highly variable startup times, a startupProbe can be used. If defined, all liveness and readiness probes are disabled until the startup probe succeeds, preventing premature killing or routing of traffic.

Effective use of readiness probes requires adherence to several key design principles.

• Check External Dependencies Lightly: The probe endpoint should verify critical dependencies (e.g., database, cache) but must be low-cost and fast. Avoid complex logic that consumes significant resources or has its own dependencies.
• Implement a Dedicated Endpoint: Use a lightweight, dedicated health endpoint (e.g., /health/ready) separate from the main application logic and the liveness probe endpoint. This allows for independent management.
• Make it Fail for Partial Health: The probe must fail if the pod is functionally impaired, even if the process is running. For example, if a connection to a required backend service is lost, the pod should stop receiving traffic.
• Avoid Over-reliance on External Systems: While checking a primary database is reasonable, a readiness probe that fails due to a transient outage in a secondary, non-essential system can cause unnecessary cascading failures. Implement graceful degradation in the probe logic.
• Combine with Liveness Probes: Use readiness probes in conjunction with liveness probes. A liveness probe checks if the container is alive (restart if dead). A readiness probe checks if it is ready for work (stop sending traffic if not).

Misconfigured readiness probes are a frequent source of deployment and runtime issues in Kubernetes.

• Too Short initialDelaySeconds: Causes the probe to start before the app is listening, marking the pod as Unready immediately and potentially causing restart loops if confused with liveness.
• Probe Logic Too Heavy: An /health endpoint that runs full integration tests will consume excessive CPU and slow the application, creating a self-inflicted denial-of-service.
• Mixing Readiness with Liveness Logic: Using the same endpoint and logic for both probes is an anti-pattern. A heavy liveness check can kill a healthy but busy pod, while a lightweight readiness check might keep a broken pod in service.
• Ignoring Transient Failures: Setting failureThreshold: 1 can cause pods to flap in and out of service due to brief network glitches or garbage collection pauses.
• Forgetting the Probe Altogether: Without a readiness probe, the pod is added to service endpoints as soon as the container process starts, which can lead to user-visible errors during initialization.
• Not Testing Probe Failure: Engineers often test the happy path but fail to validate that the pod correctly removes itself from service when its dependencies are severed.

A Kubernetes health check that determines if a container is running and responsive. Unlike a readiness probe, which checks if an app is ready for traffic, a liveness probe checks if it is alive. If it fails, Kubernetes restarts the container.

• Purpose: Detect and recover from a hung or dead process.
• Common Checks: Simple HTTP endpoint response, TCP socket connection, or command execution.
• Key Difference: A failing liveness probe triggers a restart; a failing readiness probe only removes the pod from service load balancers.

A Kubernetes health check used for applications with long initialization periods. It delays the activation of liveness and readiness probes until the application has successfully started.

• Use Case: Legacy applications, Java apps with slow JVM startup, or services that must load large data sets.
• Function: Prevents Kubernetes from killing a container via the liveness probe before it has had time to start.
• Configuration: Typically has a higher failureThreshold and periodSeconds than other probes.

A dedicated URL or API endpoint (e.g., /health or /status) exposed by a service that returns a standardized payload indicating its operational state. This is the mechanism that probes like readiness and liveness typically call.

• Response: Includes HTTP status code (200 for healthy, 5xx for unhealthy) and often a JSON body with component statuses (database, cache, etc.).
• Standardization: Frameworks like Spring Boot Actuator or K8s ecosystem tools provide built-in health endpoints.
• Depth: Can be shallow (app server only) or deep (checks all critical downstream dependencies).

A resiliency design pattern that prevents an application from repeatedly trying to execute an operation that is likely to fail. It acts as a proxy for operations, monitoring for failures and "opening" the circuit to fail fast after a threshold is crossed.

• States: Closed (normal operation), Open (failing fast, no requests passed), Half-Open (testing if the fault is resolved).
• Purpose: Prevent cascading failures and resource exhaustion (e.g., thread pool depletion).
• Relation to Probes: While a readiness probe checks internal state, a circuit breaker protects against external dependency failures. Libraries like Resilience4j and Hystrix implement this pattern.

A system design principle where functionality is reduced in a controlled, deliberate manner when a partial failure occurs. The system maintains core operations while non-essential features are temporarily disabled.

• Example: An e-commerce site disables product recommendations during a cache failure but keeps the shopping cart and checkout functional.
• Implementation: Often uses feature flags and fallback logic triggered by health check statuses or circuit breaker states.
• Goal: Preserve user experience and core business logic during outages, rather than presenting a complete failure.

A deployment and release strategy where a new version of a service is released to a small, controlled subset of users or traffic. Its health and performance metrics are compared against the stable baseline version before a full rollout.

• Process: Traffic is split between the old (stable) and new (canary) versions. Automated analysis checks error rates, latency, and business metrics.
• Relation to Health Checks: Readiness and liveness probes are critical for canary instances. If their health checks fail, the canary is automatically rolled back, preventing a bad release from affecting all users.
• Tooling: Supported by platforms like Argo Rollouts, Flagger, and Spinnaker.