Liveness Probe

A liveness probe can be configured using one of three primary mechanisms, each suited to different application architectures:

• HTTP GET Probe: The kubelet sends an HTTP request to a specified path and port on the container. A success code (2xx or 3xx) indicates liveness. This is common for RESTful vector database APIs.
• TCP Socket Probe: The kubelet attempts to open a TCP connection to a specified port on the container. Success is simply establishing the connection, useful for non-HTTP services.
• Exec Probe: The kubelet executes a specified command inside the container. An exit code of 0 indicates success. This allows for custom, complex health-check logic.

The behavior of a liveness probe is finely tuned through several key parameters in the pod specification:

• initialDelaySeconds: The number of seconds to wait after the container starts before initiating probes. Critical for applications with long boot times.
• periodSeconds: How often (in seconds) to perform the probe. Typical values range from 5 to 30 seconds.
• timeoutSeconds: Number of seconds after which the probe times out. Must be less than periodSeconds.
• successThreshold: Minimum consecutive successes for the probe to be considered successful after a failure. For liveness, this is typically 1.
• failureThreshold: The number of consecutive probe failures required for Kubernetes to consider the container unhealthy and restart it. The default is 3.

When a liveness probe consecutively fails the failureThreshold number of times, Kubernetes takes a definitive corrective action:

• The container is terminated using a SIGTERM signal, followed by SIGKILL after its grace period.
• The pod's restartPolicy (Always, OnFailure, Never) dictates the next step. For stateful services like vector databases, Always or OnFailure is typical.
• The pod enters a CrashLoopBackOff state if it repeatedly fails and restarts, with exponentially increasing delays between restarts to prevent resource consumption storms.

It is critical to differentiate liveness from readiness probes, as they serve complementary but distinct purposes:

• Liveness Probe: Answers "Is the container alive?" A failure results in a container restart. It addresses a deadlocked or unresponsive process.
• Readiness Probe: Answers "Is the container ready to serve traffic?" A failure results in the pod being removed from Service endpoints. It addresses temporary unavailability during startup or heavy load. A vector database pod should typically use both: a readiness probe to prevent traffic during index loading, and a liveness probe to restart a pod that has crashed internally.

Implementing a liveness probe for stateful, data-intensive services like vector databases requires careful design to avoid destructive restarts:

• The probe endpoint must be lightweight and local, checking core process health without triggering expensive operations (e.g., avoid a full vector scan).
• It must account for legitimate heavy load; a slow response under query pressure should not necessarily cause a restart. Tune timeoutSeconds and failureThreshold accordingly.
• For clustered databases, the probe should check the local node's health within the cluster, not the cluster's global state, to avoid unnecessary restarts of a functioning node experiencing network partition.

Liveness probe activity is integral to system observability and appears in several diagnostic locations:

• Pod Events: Probe failures generate Kubernetes events viewable via kubectl describe pod <pod-name>, showing reasons like "Liveness probe failed".
• Pod Status: The container status in kubectl get pod output shows restarts and ready state.
• Container Logs: Application logs should be inspected to understand why the health check failed (e.g., deadlock, resource exhaustion).
• Metrics: The kubelet exposes metrics like prober_probe_total and prober_probe_failed_total which can be scraped by Prometheus for cluster-wide probe failure rates.

The most common implementation, where the kubelet sends an HTTP GET request to a specified path and port on the container. A successful response (HTTP status code between 200 and 399) indicates the container is alive.

• Path: Typically a lightweight endpoint like /healthz or /live.
• Configuration: Defined by httpGet in the pod spec, with fields for path, port, and optional httpHeaders.
• Use Case: Ideal for vector database pods that expose a REST or HTTP API, allowing a quick check that the web server process is responsive.

The kubelet attempts to open a TCP connection to a specified port on the container. Success is based solely on the ability to establish a connection; no data is sent or received.

• Configuration: Defined by tcpSocket with a port field in the pod spec.
• Use Case: Perfect for vector databases that use proprietary binary protocols (e.g., gRPC, custom TCP) or where an HTTP server is unnecessary overhead. It verifies the main service port is listening.

The kubelet executes a specified command inside the container's primary process namespace. A zero exit code indicates success.

• Configuration: Defined by exec with a command array in the pod spec.
• Example Command: ["pgrep", "-f", "qdrant"] or a custom script that checks internal process health.
• Use Case: Provides maximum flexibility for complex health logic, such as checking if a specific daemon process is running or verifying internal latch states. Use with caution due to resource overhead.

Probe behavior is finely tuned through several key parameters in the pod specification:

• initialDelaySeconds: Wait time after container start before initiating probes. Critical for vector databases with long boot/initialization times.
• periodSeconds: How often to perform the probe (e.g., every 10 seconds).
• timeoutSeconds: Number of seconds after which the probe times out.
• successThreshold: Consecutive successes required to mark a failed container as healthy.
• failureThreshold: Consecutive failures required to mark a healthy container as unhealthy, triggering a restart.

A liveness probe answers "Is the process running?" A failed probe causes the container to restart.

A readiness probe answers "Is the process ready to serve traffic?" A failed probe removes the pod's IP from Service endpoints.

Key Distinction:

• A vector database pod can be live (process running) but not ready (still loading indexes into memory, replicating data).
• Use both probes together: a liveness probe with a strict check on core process health, and a readiness probe that checks for full operational capability.

Modern vector databases like Weaviate, Qdrant, and Milvus expose dedicated HTTP endpoints for these probes.

• Liveness Endpoint: Often a simple, process-internal check (e.g., Weaviate's /v1/.well-known/live).
• Implementation: The probe configuration in the Kubernetes Deployment or StatefulSet manifest points to this endpoint.
• Best Practice: The endpoint should check the core process state without dependencies on external services (like object storage) to avoid unnecessary restarts during transient external failures.

A Kubernetes mechanism that determines if a containerized application, such as a vector database pod, is fully initialized and ready to accept network traffic. It runs after the container starts but before the liveness probe begins periodic checks.

• Purpose: Prevents the pod from receiving requests before it has loaded its indices, connected to dependencies, or completed startup scripts.
• Key Difference from Liveness: A failed readiness probe does not restart the pod; it only removes the pod's IP from the service endpoint list until it passes.
• Common Checks: Verifying connection to a persistent volume, ensuring the vector index is loaded into memory, or confirming a TCP socket is listening.

A dedicated API endpoint (e.g., /health) exposed by a vector database service that returns the operational status of the system. This is the programmatic interface that liveness and readiness probes call.

• Implementation: Typically returns a simple HTTP 200 OK response with a JSON payload containing status details for subsystems (e.g., {"index_status": "loaded", "memory_usage": "85%"}).
• Lightweight vs. Deep Checks: A liveness probe should call a lightweight endpoint that checks if the process is alive. A readiness probe may call a more expensive endpoint that verifies backend connectivity.
• Best Practice: The endpoint should have minimal dependencies to avoid cascading failures; it should not, for instance, perform a full vector search as part of the check.

A stability pattern implemented in client applications or service meshes that temporarily stops calling a failing dependency (like a vector database API) after a threshold of failures is reached.

• Mechanism: It operates in three states: Closed (normal requests), Open (requests fail immediately, allowing the downstream service to recover), and Half-Open (allowing a test request to see if the service is healthy again).
• Relation to Probes: While a liveness probe restarts a local container, a circuit breaker protects a remote service from being overwhelmed by repeated failing requests from clients. It is a client-side complement to server-side health checks.
• Use Case: If a vector database's health check starts failing due to high load, a circuit breaker in calling services can prevent a retry storm and give the database time to recover.

A deployment strategy for vector database clusters where nodes are restarted one at a time in a controlled sequence, allowing the service to remain available with minimal disruption.

• Process: The orchestrator (e.g., Kubernetes) terminates a pod, waits for its replacement pod to pass its readiness probe, and then moves on to restart the next pod.
• Dependency on Probes: This strategy relies entirely on correctly configured liveness and readiness probes. Without them, the orchestrator cannot determine if a pod is healthy before proceeding, risking simultaneous failures.
• Benefit: Enables zero-downtime upgrades of vector database software, security patches, or node configuration changes.

The automatic process of switching operations from a failed primary node in a distributed vector database cluster to a healthy standby replica to maintain service availability.

• Trigger: Often initiated by a consensus system (like Raft) or an external monitor that detects the primary node is unresponsive—a state ultimately determined by a form of liveness detection.
• Probe's Role: In a Kubernetes-managed cluster, the liveness probe on the primary pod may fail, causing a restart. If restarts fail repeatedly, a higher-level cluster manager may initiate a failover to a different pod or node.
• Objective: Minimizes Recovery Time Objective (RTO) by automating the response to node failure.

A target level of reliability for a vector database service, formally defined as a measurable metric like uptime percentage or query latency. Liveness probes are a primary tool for measuring and enforcing availability SLOs.

• Example SLO: "The vector search API must be available 99.9% of the time over a 30-day period."
• Measurement: Each time a liveness probe fails and causes a restart, it contributes to downtime, consuming the system's error budget.
• Configuration Impact: Aggressive liveness probe timeouts (e.g., 1 second) may cause unnecessary restarts and SLO violations during brief load spikes. Conservative timeouts may leave unresponsive pods running, also violating SLOs. The probe configuration is a direct engineering lever for SLO adherence.