Health Check

Health checks are typically implemented as two distinct probe types in containerized environments like Kubernetes.

• Liveness Probe: Determines if the service process is running. A failure triggers a container restart.
• Readiness Probe: Determines if the service is ready to accept traffic (e.g., model loaded, dependencies connected). A failure removes the pod from the load balancer. For AI services, a readiness probe might check that the model weights are loaded into GPU memory and that the vector database connection is active.

A health check is implemented as a dedicated, lightweight HTTP endpoint (e.g., /health).

• A 200 OK response with a simple JSON payload ({"status": "healthy"}) indicates operational health.
• Any 4xx or 5xx status code signals failure. For AI services, the endpoint logic should verify critical subsystems: the model inference engine, tokenizer, and any retrieval or memory backends (e.g., vector database). The check must be low-latency to avoid becoming a performance bottleneck itself.

Health check failures directly impact Service Level Indicators (SLIs) like error rate and availability, which are measured against Service Level Objectives (SLOs).

• A failing readiness probe that causes a pod to be taken out of service may protect the overall error budget by preventing bad requests.
• Persistent liveness probe failures that cause frequent restarts consume the error budget through increased downtime. Health check status is a foundational signal for calculating SLO burn rate and triggering alerts before user impact becomes severe.

Beyond process status, health checks for AI services must validate model-specific states:

• Model Load Status: Confirms the serialized model is loaded and accessible in memory/GPU.
• Context Window Availability: For LLMs, checks that the request queue or continuous batching system is not saturated.
• Tool/API Connectivity: For agentic systems, verifies connections to external tools and APIs defined in the Model Context Protocol.
• Retrieval System Latency: Probes the vector database or knowledge graph to ensure retrieval SLIs (e.g., p95 latency) are within acceptable bounds for RAG systems.

Health check behavior is governed by three key parameters:

• periodSeconds: How often to perform the probe (e.g., every 10 seconds).
• timeoutSeconds: Time to wait for a response before considering it a failure (must be less than the period).
• failureThreshold: The number of consecutive failures required to declare the container unhealthy. These settings must be tuned for AI workloads. A long model inference latency might require a longer timeout for a readiness check that runs a tiny inference to validate the pipeline.

Advanced health checks simulate real user behavior via synthetic transactions. For an AI service, this could be:

• Sending a canonical query to a RAG system and validating the response contains citations.
• Executing a simple tool-calling sequence with an autonomous agent.
• Measuring Time to First Token (TTFT) and Time Per Output Token (TPOT) for a streaming LLM endpoint. These checks validate the entire Critical User Journey (CUJ) and provide a stronger signal than a simple endpoint ping, closely aligning health with user-centric SLOs.

A liveness probe determines if a container or service process is running. For an AI model server (e.g., a TensorFlow Serving or vLLM instance), this is a simple HTTP GET request to a designated /health/live endpoint.

• Purpose: Signals to the orchestrator (Kubernetes) that the process has not crashed and should be restarted if unhealthy.
• Implementation: Often a lightweight check that the server's HTTP stack is responsive, without invoking the model.
• Example: curl -f http://model-service:8000/health/live returns HTTP 200 if the server process is alive.

A readiness probe assesses if a service is ready to accept user traffic. For AI services, this must confirm the model is loaded into memory and the inference engine is initialized.

• Purpose: Prevents traffic from being routed to a pod that is booting up or a model that is still loading.
• Critical Check: Verifies GPU availability and that the model's computational graph is cached. A failed readiness probe tells the load balancer to stop sending requests.
• Implementation: A call to a /health/ready endpoint that performs a trivial inference (e.g., on a zero tensor) to validate the full pipeline.

Complex AI services like Retrieval-Augmented Generation (RAG) have critical downstream dependencies. A comprehensive health check must validate each component.

• Typical Dependencies: Vector database (e.g., Pinecone, Weaviate), embedding model endpoint, and the core LLM provider.
• Implementation Pattern: The health check endpoint performs a lightweight query to each dependency:
  • A ping to the vector database cluster.
  • A simple embedding generation for a test string.
  • A tokenization call to the LLM API.
• Failure Action: If any dependency is unhealthy, the service marks itself as not ready, preventing partial failures for user requests.

Beyond basic uptime, health checks can validate model performance and output quality to catch silent degradations before users are impacted.

• Example Checks:
  • Output Schema Validation: Generate a response to a canned prompt and validate the JSON structure against a predefined schema.
  • Numerical Stability: Run a fixed inference with a random seed and assert the output logits or embeddings are within an expected numerical range.
  • Hallucination Baseline: For a RAG system, query with a known fact and assert the answer contains a required keyword or entity.
• Frequency: These deeper checks run less frequently (e.g., every 30 seconds) than simple liveness probes (every 2 seconds) due to higher computational cost.

A health check can monitor inference performance against internal Service Level Indicators (SLIs) to detect infrastructure degradation.

• How it works: The probe executes a standardized, representative inference request and measures the latency.
• Alerting: If the p95 latency for the probe exceeds a threshold (e.g., 500ms for a simple task), the health check fails. This can indicate:
  • GPU thermal throttling.
  • Memory contention from other processes.
  • Network latency spikes to a remote model host.
• Throughput Check: Can also verify the service accepts a small burst of concurrent requests without queueing errors.

Health check results are integrated with client-side circuit breakers (e.g., using libraries like Resilience4j or Polly) to prevent cascading failures.

• Mechanism: The client library tracks the failure rate of recent requests to an instance. If failures exceed a threshold, the circuit opens and all subsequent requests fail fast for a cooldown period.
• Health Check Role: After the cooldown, a single health check request is sent as a half-open test. If it succeeds, the circuit closes and normal traffic resumes. If it fails, the circuit re-opens.
• Benefit: This pattern protects the overall system by isolating unhealthy model instances, allowing them time to recover or be replaced by the orchestrator.

A Service Level Objective (SLO) is a quantitative target for the reliability, performance, or quality of a service, expressed as a percentage of requests that must meet a specific Service Level Indicator (SLI) over a defined time window. For AI services, SLOs are critical for defining acceptable performance bounds.

• Example: "99.9% of inference requests must have a latency under 100ms over a 30-day window."
• SLOs are internal goals, distinct from external Service Level Agreements (SLAs) which carry contractual penalties.
• Setting appropriate SLOs involves balancing user experience with engineering feasibility and cost.

A Service Level Indicator (SLI) is a directly measurable metric that quantifies a specific aspect of a service's performance. It is the raw measurement that an SLO is based upon. For AI systems, SLIs must be carefully chosen to reflect user-perceived quality.

• Common AI SLIs: Model inference latency, error rate (e.g., 5xx HTTP errors), throughput (requests per second), and task-specific quality metrics like answer faithfulness or retrieval precision.
• An SLI is always defined with a measurement method and aggregation window (e.g., average latency over 1 minute).
• Effective SLIs are user-centric, measuring outcomes that directly impact the end-user experience.

An error budget is the allowable amount of service unreliability, calculated as 100% - SLO. It defines the risk a team can accept for deploying new features or making changes without violating the SLO.

• If an SLO is 99.9% reliability, the error budget is 0.1% unreliability over the compliance period.
• The budget can be spent on planned risk (like deployments) or consumed by unplanned incidents.
• Burn rate monitors how quickly the error budget is being consumed, triggering alerts if exhaustion is imminent. This shifts focus from "is something broken?" to "are we at risk of breaking our promises?"

A golden signal is one of four fundamental metrics used in Site Reliability Engineering (SRE) to comprehensively monitor the health and performance of any service: Latency, Traffic, Errors, and Saturation.

• Latency: The time it takes to service a request (e.g., p95 inference time).
• Traffic: A measure of demand (e.g., queries per second, token generation rate).
• Errors: The rate of failed requests (e.g., non-2xx HTTP statuses, model hallucination rate).
• Saturation: How "full" the service is (e.g., GPU utilization, queue depth). These signals provide a holistic view beyond simple uptime and are the primary sources for defining SLIs.

A canary deployment is a release strategy where a new version of a service (e.g., an updated ML model) is deployed to a small, representative subset of users or traffic. Its performance is closely monitored against SLIs before a full rollout.

• This technique is essential for validating SLO compliance of new changes in a low-risk, production-like environment.
• Production Canary Analysis involves comparing the canary's SLI performance (latency, error rate) against a stable baseline.
• If the canary violates error budget policies, the deployment is automatically rolled back, preventing a widespread SLO breach.

Tail latency, measured by percentiles like p95 or p99, represents the maximum latency experienced by the slowest 5% or 1% of requests. It is critical for AI services because user perception is often defined by worst-case performance.

• A p99 latency of 500ms means 99% of requests are faster than 500ms, and 1% are slower.
• Tail Latency Amplification in distributed AI systems can cause p99 to be orders of magnitude worse than p50 due to queuing, garbage collection, or dependency chains.
• SLOs for user-facing AI features must explicitly target tail latency percentiles, not just averages, to ensure a consistently good experience.