Readiness Probe

The primary function is to act as a traffic gatekeeper. It prevents the Kubernetes service from routing requests to a pod until the probe succeeds. This is critical for vector databases, which must:

• Load large vector indices into memory.
• Establish connections to persistent storage.
• Complete any schema migrations or recovery processes. A pod only enters the Ready state and receives production traffic after its readiness probe passes.

Kubernetes supports three primary probe mechanisms, each suitable for different initialization checks:

• HTTP GET Probe: Checks a specific endpoint (e.g., /ready). Returns success on a 2xx or 3xx HTTP status code. Ideal for checking if the database's HTTP API is listening.
• TCP Socket Probe: Attempts to open a TCP connection on a specified port. Useful for verifying the core database port (e.g., for gRPC or proprietary protocols) is open.
• Exec Probe: Executes a command inside the container. Success is a zero exit code. Can run complex scripts to verify internal state, like checking if an index build is complete.

Probe behavior is finely tuned via several key parameters in the pod specification:

• initialDelaySeconds: Wait time after container start before initiating probes. Crucial for allowing slow-starting processes (e.g., index loading) to begin.
• periodSeconds: How often to perform the probe (e.g., every 10 seconds).
• timeoutSeconds: Time after which a probe is considered failed if no response.
• successThreshold: Consecutive successes required to mark a pod as ready.
• failureThreshold: Consecutive failures required to mark a pod as Not Ready and remove it from service endpoints.

It is essential to differentiate readiness probes from liveness probes:

• Readiness Probe: Answers, "Can this pod handle traffic?" Failure results in the pod being removed from service load balancers, but the container is not restarted.
• Liveness Probe: Answers, "Is this pod alive?" Failure results in the container being killed and restarted by the kubelet. A vector database pod can be alive (liveness probe passing) but not ready (readiness probe failing) during maintenance or heavy load, allowing for graceful handling without restarts.

Readiness probes are fundamental to zero-downtime deployments and cluster stability:

• During a rolling update, new pods are added to the replica set only after their readiness probe passes, ensuring they can serve traffic before old pods are terminated.
• If a pod becomes overloaded or enters a degraded state (e.g., cache thrashing), a failing readiness probe triggers its removal from the service, implementing automatic load shedding. This protects the pod and prevents client timeouts, allowing it to recover.
• This enables patterns like canary releases where a new version is deployed to a subset of pods, with traffic directed only to those that report ready.

Effective readiness probes for vector databases must check deep application state, not just superficial availability:

• Index Health: Verify that the primary vector index (e.g., HNSW, IVF) is fully loaded and queryable.
• Dependency Connectivity: Confirm connections to required services are active (e.g., object storage for persisted vectors, metadata databases).
• Resource Saturation: Check that in-memory caches are populated above a minimum threshold to avoid cold start latency for all incoming queries.
• Write-Ahead Log (WAL) Recovery: Ensure any necessary recovery from the WAL has completed after a restart. A poorly designed probe that passes too early can lead to timeouts and corrupted client operations.

A Kubernetes mechanism that periodically checks if a containerized application, such as a vector database pod, is running and responsive. Unlike a readiness probe, its failure triggers a container restart. It answers the question: "Is the process alive?"

• Purpose: Detect and recover from deadlocks or hung processes.
• Common Checks: Simple TCP socket connection or HTTP endpoint response.
• Key Difference: A liveness probe failure restarts the pod; a readiness probe failure only stops routing traffic to it.

A dedicated API endpoint (e.g., /health) on a vector database service that returns the operational status of the system. This endpoint is the target for both liveness and readiness probes from orchestration platforms.

• Typical Responses: HTTP 200 OK with a JSON payload detailing component status (e.g., {"index_loaded": true, "wal_ready": true}).
• Implementation: Should check critical dependencies like the vector index state, write-ahead log health, and connection pools.
• Best Practice: The readiness probe should check a more comprehensive set of conditions than the liveness probe.

The maximum acceptable duration of downtime for a system, defining the target time within which operations must be restored after a failure. For a vector database, this drives architectural decisions around failover automation and backup strategies.

• Business Metric: Dictates the required speed of failover processes.
• Engineering Implication: An RTO of seconds necessitates hot standbys and automated orchestration. An RTO of hours may allow for manual restoration from backups.
• Relation to RPO: RTO (time to restore) is often considered alongside Recovery Point Objective (RPO) (amount of data loss acceptable).

The automatic process of switching operations from a failed primary node in a distributed vector database cluster to a healthy standby replica. The goal is to maintain service availability with minimal disruption.

• Triggered By: Failure of liveness/readiness probes, network partition, or hardware fault.
• Mechanism: Often managed by a cluster coordinator or external orchestrator (e.g., Kubernetes).
• Post-Failover: Client connections are redirected to the new primary. A subsequent failback process may return traffic to the original node once repaired.

A stability pattern that temporarily stops calling a failing downstream service after a threshold of failures is reached. In vector database operations, this can protect the database from a cascading failure caused by a malfunctioning embedding model API.

• States: Closed (normal operation), Open (requests fail immediately), Half-Open (testing if dependency has recovered).
• Use Case: If an embedding service times out, the circuit breaker opens, causing ingestion requests to fail fast rather than pile up and exhaust database connections.
• Benefit: Allows the failing dependency time to recover and prevents resource exhaustion.

A target level of reliability for the accuracy of a vector database's similarity search. It is formally defined as the proportion of true nearest neighbors successfully returned over a measurement period (e.g., "99.9% recall over 30 days").

• Recall: In ANN search, the fraction of true top-k nearest neighbors found by the approximate index.
• Operational Impact: An SLO defines what "ready for traffic" means for search quality. A probe could check if recall degradation is detected.
• Error Budget: The permissible deviation from the SLO, used to govern the pace of risky changes like index algorithm updates.