Vector Durability

The practice of forcing the operating system to flush write buffers to the physical storage medium immediately, rather than caching them in volatile memory. For vector databases, this is often controlled by flags like O_SYNC or the fsync() system call. While this increases write latency, it provides the strongest guarantee that data is permanently stored, making it critical for durability over pure performance in certain enterprise scenarios.

The process of creating and maintaining redundant copies of vector data across multiple independent storage nodes or geographical regions. This provides durability against hardware failures. Common strategies include:

• Synchronous Replication: The write is confirmed only after all replicas acknowledge it, guaranteeing strong consistency and durability at the cost of higher latency.
• Asynchronous Replication: The write is confirmed locally first, then propagated to replicas, offering lower latency but a small window of potential data loss if the primary node fails.

A data protection method that provides high durability with lower storage overhead than simple replication. The original vector data is broken into k data fragments, encoded into n total fragments (where n > k), and distributed across multiple nodes. The original data can be reconstructed from any k fragments. This allows the system to tolerate the failure of n-k nodes. For example, a 10-of-16 scheme can survive 6 node failures while adding only 60% storage overhead, compared to 300% for 3x replication.

The process of creating a read-only, crash-consistent copy of the entire vector index and associated metadata at a specific moment. Snapshots are typically stored on durable, cost-effective object storage (e.g., S3, GCS). This mechanism enables:

• Disaster Recovery: Full restoration of the database from a known good state.
• Data Versioning: Rolling back to a previous state in case of corruption or erroneous operations.
• Safe Index Rebuilding: Creating a new index from a snapshot while the live system continues to serve queries.

A core durability mechanism where all modifications to vector data are first written as sequential entries to a persistent, append-only log before being applied to the main index. This ensures that in the event of a system crash, the database can recover the exact state by replaying the log. WAL is fundamental for providing Atomicity and Durability (the 'A' and 'D' in ACID) for vector operations.

The process of creating and maintaining redundant copies of vector data across multiple storage nodes or geographical regions. This provides fault tolerance and high availability. Common strategies include:

• Synchronous Replication: Writes are confirmed only after data is persisted on multiple replicas, guaranteeing strong consistency.
• Asynchronous Replication: Writes are confirmed after the primary node, with replicas updated later, offering lower latency but potential for temporary data loss. Replication is a primary method for achieving durability against node or data center failures.

An adaptation of the Log-Structured Merge-Tree storage architecture optimized for high-throughput ingestion of vectors. It batches writes in a memory-based memtable and flushes them to immutable, sorted files on disk called SSTables (Sorted String Tables). Periodic compaction merges these files. This design is inherently durable as writes are quickly persisted to sequential disk writes and is central to the storage engines of databases like RocksDB, which underpin many vector databases.

A data protection method that provides high durability with lower storage overhead than simple replication. The vector dataset is broken into k data fragments, encoded into n total fragments (where n > k), and distributed across nodes. The original data can be reconstructed from any k fragments. This allows the system to survive the loss of n-k nodes. For example, a 10-of-16 encoding can lose 6 nodes but only uses 1.6x storage overhead, compared to 3x for triple replication.

An architectural pattern that automatically moves vector data between storage tiers based on access patterns and cost policies, while maintaining overall durability guarantees. A common lifecycle is:

1. Hot Tier (SSD/NVMe): For frequently accessed, newly ingested vectors.
2. Warm/Cold Tier (HDD): For less frequently queried historical vectors.
3. Archival Tier (Object Storage): For compliance or long-term backup. Durability policies (e.g., replication factor, erasure coding) are often defined per tier.

The formal guarantee provided by a distributed vector database regarding when a written vector becomes visible to subsequent read operations across different replicas. This directly impacts the durability semantics perceived by the client. Key models include:

• Strong Consistency: A read is guaranteed to see the most recent write.
• Eventual Consistency: Replicas will converge to the same state given no new writes.
• Causal Consistency: Preserves cause-and-effect order of operations. Durability (data survival) is distinct from consistency (data visibility timing).