Rolling Restart

The core mechanism of a rolling restart is the sequential, one-by-one restart of nodes in a cluster. The orchestrator (e.g., Kubernetes, a custom scheduler) follows a defined order:

• Drains a node, preventing new queries from being routed to it.
• Waits for in-flight operations to complete.
• Stops the vector database process on that node.
• Applies updates (new binary, config).
• Restarts the process and runs health checks.
• Marks the node as ready and returns it to the load balancer. Only after one node is fully healthy does the process begin on the next node. This ensures the cluster's overall shard replication factor is never fully compromised.

The primary goal is zero-downtime for client applications. Because at least (N-1) nodes in an N-node cluster are always operational:

• Read queries continue to be served by online replicas.
• Write operations are accepted as long as the required consistency level (e.g., writes acknowledged by 2 out of 3 replicas) can be met by the remaining online nodes.
• The load balancer automatically redirects traffic away from the node being restarted. This is in stark contrast to a full cluster restart, which causes a total service outage.

Rolling restarts are rarely manual. They are automated and integrated with orchestration platforms:

• Kubernetes: Managed via a Deployment or StatefulSet strategy with maxUnavailable: 1. The kubelet uses liveness and readiness probes to determine node health.
• Cloud Managed Services: AWS MemoryDB for Redis Vector, Pinecone, and Weaviate Cloud manage this transparently during patching.
• Custom Schedulers: In-house systems use cluster metadata to decide restart order, often prioritizing follower replicas before primary nodes to minimize write disruption.

Vector databases are stateful; each node holds a shard of the vector index. The restart process must preserve data integrity:

• Persistent Volumes: Node storage is typically backed by persistent volumes (PVs) in Kubernetes, so the restarted pod reattaches to the same index data.
• Index Warm-up: After restart, the in-memory vector index (e.g., HNSW, IVF) may need to be paged in from disk, causing temporary cold start latency for queries on that node.
• Write-Ahead Log (WAL) Replay: On startup, the node replays its WAL to ensure all committed writes are applied, maintaining consistency with other replicas.

Each step is gated by verification to prevent cascading failure:

• Pre-flight Checks: Cluster health is verified before beginning the sequence.
• Post-restart Health Checks: The node must pass its readiness probe (e.g., responding to a simple query) before being reintroduced to the pool.
• Automated Rollback: If a node fails to start or fails its health checks, the process halts. The orchestrator may attempt to restart the old version on the failed node and alert operators, preventing a defective update from taking down the entire cluster.

While availability is maintained, performance is temporarily degraded:

• Increased Latency: The cluster operates with reduced capacity. Query load is distributed among fewer nodes, potentially increasing p95 and p99 latency.
• Cache Disruption: The restarted node's vector cache is cleared, resulting in a lower cache hit ratio and more disk I/O for its first queries.
• Planning Consideration: Restarts are often scheduled during maintenance windows or periods of low traffic. The strategy is contrasted with blue-green deployment, where a full parallel cluster is switched over instantly, avoiding this performance penalty but at higher infrastructure cost.

A release strategy where two identical production environments (blue and green) run simultaneously. Traffic is routed entirely to one environment (e.g., blue). After deploying a new version to the idle environment (green) and validating it, traffic is switched over instantaneously. This enables zero-downtime deployments and immediate rollback by switching traffic back.

• Key Difference from Rolling Restart: A rolling restart updates a single cluster in-place, while blue-green uses two separate, complete environments.
• Use Case: Major version upgrades or architectural changes where a clean, atomic cutover is preferred over a sequential node update.

A deployment technique where a new software version is gradually exposed to a small, controlled subset of users or traffic before a full rollout. In a vector database cluster, this might mean updating a single node and directing a fraction of query traffic to it for monitoring.

• Purpose: To detect performance regressions, bugs, or recall issues with minimal impact.
• Progression: If metrics (e.g., latency, recall, error rates) remain within Service Level Objectives (SLOs), the update is rolled out to more nodes. It is often used in conjunction with a rolling restart for the broader rollout phase.

The automatic process by which a distributed system switches operations from a failed primary node to a healthy standby replica. In a vector database, this maintains availability for read and write operations during a node failure.

• Relation to Rolling Restart: A rolling restart proactively manages node health to prevent unplanned failovers. During the restart, the cluster's replication and leader election mechanisms ensure a form of controlled, planned failover for each node being cycled.
• Mechanism: Often relies on a consensus protocol (like Raft) to elect a new leader and update the cluster's metadata.

Kubernetes health checks used to manage container lifecycle. They are critical for automating rolling restarts safely.

• Liveness Probe: Determines if a pod (e.g., a vector database node) is running. If it fails, Kubernetes restarts the container.
• Readiness Probe: Determines if a pod is ready to serve traffic. If it fails, the pod is removed from the service load balancer.

During a rolling restart, the orchestrator:

1. Updates a pod's configuration/image.
2. Waits for its readiness probe to pass before considering the node 'ready'. This ensures the node's index is fully loaded (cold start latency has passed) and it can serve queries.
3. Only then moves to restart the next pod.

A scheduled period of time during which planned, disruptive operations are performed on a system. While a rolling restart is designed to minimize disruption, some operations may still require a formal maintenance window.

• Operations requiring a window: Changes that are not node-by-node, such as schema migrations that require a full cluster lock, or changes to underlying shared storage.
• Operations suitable for rolling restart: Software patches, configuration changes, and node resource updates that can be applied sequentially without a full service halt. A rolling restart effectively eliminates the need for a maintenance window for these types of changes.

A configurable setting in distributed databases that defines how many replicas must acknowledge a read or write operation before it is considered successful. It balances between data accuracy (strong consistency) and latency (weak consistency).

• Impact on Rolling Restarts: During a node restart, it is temporarily unavailable. The chosen consistency level dictates client experience:
  • Strong Consistency (e.g., QUORUM): Writes/reads may experience higher latency or temporary errors if insufficient replicas are available.
  • Eventual Consistency: Clients may read stale data from remaining replicas during the restart.
• Best Practice: Understand your application's tolerance and configure consistency levels appropriately before executing rolling restarts.