Index State

A Write-Ahead Log (WAL) is a fundamental durability mechanism. All data modification operations (inserts, updates, deletes) are first recorded as entries in a persistent, append-only log file before the changes are applied to the main in-memory or on-disk vector index. This ensures:

• Crash Recovery: After a system failure, the database can replay the WAL to reconstruct the last consistent state.
• Atomicity: Operations are guaranteed to be fully recorded or not at all.
• Replication: The log sequence provides an ordered record for syncing data to replicas in distributed systems.

Consistency Level defines the visibility guarantee for read and write operations in a distributed vector database. It governs the trade-off between data freshness, availability, and latency. Common levels include:

• Strong Consistency: A read is guaranteed to return the most recent write. This can increase latency.
• Eventual Consistency: Writes will propagate to all replicas eventually, but reads may temporarily return stale data, offering higher availability.
• Session Consistency: Guarantees that a client will see its own writes within a session. Configuring this is critical for applications balancing search accuracy with system resilience during network partitions.

Snapshot Isolation is a transaction isolation level that provides a query with a consistent, read-only view of the database state as it existed at a specific point in time. When a query begins, it takes a "snapshot" of the committed data. Key implications:

• Non-Blocking Reads: Long-running analytical queries do not block or get blocked by concurrent write operations (inserts, updates).
• Repeatable Reads: Within the same transaction, repeated queries see the same data, preventing phantom reads.
• Versioning: Often implemented using multi-version concurrency control (MVCC), where older versions of updated vectors are retained temporarily. This is essential for generating consistent analytics from a live index.

Incremental Indexing is a strategy for updating a vector index with new or modified embeddings without requiring a full rebuild. This is crucial for maintaining low-latency updates and high data freshness. Mechanisms vary by index type:

• Graph-based indexes (e.g., HNSW) allow direct insertion of new nodes and connections.
• Tree-based indexes may require periodic rebalancing.
• Quantization-based indexes might update codebook assignments in batches. The alternative, a full rebuild, halts queries and consumes significant resources. The choice between incremental updates and rebuilds is a core operational decision impacting index state management.

Vector Tombstoning is a deletion strategy where a vector is marked as logically deleted (a "tombstone") rather than being physically removed from the index data structure immediately. This approach provides several benefits:

• Efficient Deletes: Physical removal from complex graph or tree structures can be expensive. Tombstoning is a fast, O(1) operation.
• Consistency in Distributed Systems: Ensures all replicas agree on the deletion before cleanup.
• Query Integrity: Tombstoned vectors are filtered out of search results. A separate garbage collection process later reclaims the physical space during maintenance windows. This decouples immediate user action from costly background operations.

Data Freshness is a metric quantifying how up-to-date the information in a vector index is relative to the source data systems. It is a direct determinant of search relevance in dynamic applications. Freshness is impacted by:

• Ingestion Latency: The delay between a change at the source and its reflection in the vector index.
• Indexing Strategy: Incremental vs. batch rebuilding.
• Pipeline Health: Failures in Change Data Capture (CDC) or embedding pipelines. Low freshness leads to stale search results. Monitoring freshness—often measured as the age of the newest successfully indexed record—is a key index state health indicator.