Incremental Indexing

The foundational mechanism that monitors a source system for create, update, and delete events. Instead of scanning the entire database, CDC reads the transaction log to identify only the delta of changes. This ensures the indexing pipeline reacts to a single modified row rather than re-processing millions of static records. Common implementations include Debezium for SQL databases and watch-based triggers in cloud object storage like S3.

The atomic database operation that handles both insertion and update in a single command. When a document changes, the pipeline generates a new embedding vector. The upsert logic checks if a vector with the same document ID already exists:

• If new: The vector and metadata are inserted.
• If existing: The old vector is overwritten with the new embedding and timestamp. This prevents duplicate entries and ensures the index reflects the latest document state without manual cleanup.

A soft-delete strategy used to handle removal events in immutable vector indexes. Rather than performing an expensive physical deletion and index rebuild, a tombstone marker is written to the metadata. During retrieval, a post-filtering step excludes any vectors flagged as deleted. A background compaction process later physically removes tombstoned vectors during low-traffic windows, maintaining query speed while ensuring deleted data is never returned to the user.

A scalability pattern where the vector index is split into time-based or category-based partitions. New data is written to a small, mutable 'hot' partition, while historical data resides in larger, immutable 'cold' partitions. Incremental updates only affect the hot partition, drastically reducing the write amplification. Queries are fanned out across all partitions and results are merged, allowing the system to handle high ingestion rates without locking the entire index.

A decoupling pattern that separates content change detection from vector generation. When a source document is modified, the pipeline immediately updates the raw text in the metadata store but queues the re-embedding task. The user query interface continues to serve the old vector (with a stale flag) until the new embedding is generated and upserted. This prevents blocking I/O from slowing down the ingestion pipeline and ensures the system remains responsive under heavy write loads.

A technique to handle schema evolution without re-indexing the entire corpus. When metadata fields are added or changed, new documents are indexed with the new schema version, while old vectors retain their original structure. The retrieval layer applies a compatibility filter that normalizes both schemas at query time. This allows the index to span multiple metadata versions simultaneously, enabling continuous operation during long-running data migration processes.