Health Check

Health checks are typically bifurcated into two distinct probe types with different operational semantics:

• Liveness Probe: Answers 'Is the container running?' A failure triggers a container restart by the orchestrator (e.g., Kubernetes). It checks for deadlocks and unrecoverable crashes.
• Readiness Probe: Answers 'Can the container serve traffic?' A failure temporarily removes the pod from the service load balancer without killing it. This is critical during model loading or warming.

Misconfiguring these probes is a common cause of cascading deployment failures.

A shallow health check simply returns a 200 OK if the web server process is alive. A deep health check validates the entire critical path:

• Model Artifact Accessibility: Verifies the model weights file is present and not corrupted.
• Dependency Reachability: Pings the vector database, feature store, or external API dependencies.
• GPU Availability: Confirms the CUDA runtime is accessible and the GPU has sufficient memory.
• Inference Smoke Test: Runs a deterministic, low-latency prediction to validate the computation graph.

Deep checks provide higher confidence but consume resources; they should be called less frequently than shallow checks.

The health endpoint must adhere to a strict contract to be consumed by Kubernetes, Envoy, HAProxy, or cloud load balancers:

• HTTP Status Code: 200 for healthy, 503 Service Unavailable for unhealthy. Avoid 500 codes which can trigger ambiguous error handling.
• Response Payload: A structured JSON object containing component-level statuses (e.g., "database": "up", "model_loaded": true).
• Latency Budget: The probe handler must respond in under 1 second to prevent orchestrator timeouts.
• Idempotency: The GET request must not mutate state or trigger expensive side effects.

Large Language Models (LLMs) and diffusion models have significant initialization latency due to weight loading. A startup probe is a specialized check for slow-starting containers:

• It runs before liveness or readiness probes are activated.
• It protects the container from being killed by the liveness probe's initialDelaySeconds timeout during the lengthy model loading phase.
• Once the startup probe succeeds, it hands off control to the standard liveness/readiness probes.

Without a startup probe, orchestrators often enter a fatal crash loop during deployment.

Advanced health checks distinguish between symptoms and root causes to accelerate incident response:

• Symptom: High inference latency or 503 errors.
• Cause: The connection pool to the Redis cache is exhausted.

A well-designed health endpoint returns granular failure reasons in the response body, allowing runbook automation to trigger specific remediation scripts (e.g., resetting the connection pool) rather than a generic pod restart.

Health checks are the primary signal for the circuit breaker pattern. When a downstream AI service's health endpoint fails consecutively:

• The circuit breaker trips to an OPEN state, immediately failing requests without waiting for network timeouts.
• This prevents cascading failures and resource exhaustion in the upstream service.
• The circuit enters a HALF-OPEN state after a sleep window, allowing a limited number of probe requests to test if the downstream service has recovered.

This integration is essential for maintaining system resilience under load.