Vector Database

The fundamental capability is creating specialized indexes on vector embeddings to enable sub-linear search times. Unlike traditional databases that index scalar values, vector databases use Approximate Nearest Neighbor (ANN) algorithms to organize data points in a multi-dimensional space. Common indexing methods include:

• Graph-based (HNSW): Constructs a hierarchical navigable small world graph for fast traversal.
• Tree-based (Annoy): Builds binary trees that partition the vector space.
• Quantization-based (IVF): Uses clustering (Inverted File Index) and product quantization to compress vectors. These indexes trade a minimal, configurable amount of accuracy for orders-of-magnitude faster search compared to brute-force k-Nearest Neighbors (k-NN).

Vector databases perform similarity search by calculating the "closeness" between a query vector and indexed vectors using mathematical distance or similarity metrics. The choice of metric depends on the embedding model and use case.

• Cosine Similarity: Measures the cosine of the angle between vectors, ideal for text embeddings where magnitude is less important than direction.
• Euclidean Distance (L2): Measures the straight-line distance between points in the vector space.
• Inner Product (Dot Product): Used for Maximum Inner Product Search (MIPS), common in recommendation systems. The database efficiently computes these metrics over billions of vectors using its optimized indexes.

Beyond pure vector search, these databases support hybrid search by combining semantic similarity with structured metadata filtering. This allows for precise, context-aware queries.

• Example: "Find technical reports about battery safety published after 2023." The system filters documents by doc_type: report and date > 2023-01-01, then performs a vector search on the remaining subset using an embedding of "battery safety."
• Filtered Search: Apply conditional filters (e.g., user_id = 'abc', price < 100) before or after the vector search to ensure results meet business logic constraints.

Vector databases are built for horizontal scalability. They manage massive datasets by sharding (partitioning) vector indexes across multiple nodes and replicating data for high availability.

• Sharding: Vectors are distributed based on a sharding key (e.g., tenant ID, geographic region) or algorithmically across the cluster.
• Load Balancing: Incoming search queries are distributed across shards and processed in parallel, aggregating results for low-latency responses.
• Dynamic Scaling: Nodes can be added or removed without downtime, allowing the system to handle increasing data volume and query throughput, a critical requirement for production Retrieval-Augmented Generation (RAG) systems.

They provide full Create, Read, Update, Delete (CRUD) lifecycle management for vector data, unlike simple vector search libraries (e.g., FAISS).

• Upserts: Efficiently insert new vectors or update existing ones, with automatic index refresh.
• Deletion: Remove vectors by ID or via metadata filters, ensuring index consistency.
• Versioning & Point-in-Time Recovery: Some systems offer data versioning, allowing rollback to previous states—essential for audit trails and debugging agentic memory.
• Persistence: Guarantee durability by writing vectors and metadata to disk, surviving process restarts.

A key capability for dynamic agentic memory is supporting real-time or near-real-time index updates. As an agent learns new information, the memory store must be updated with low latency.

• Incremental Indexing: New vectors are added to the index without requiring a full rebuild, enabling memory update operations.
• Event-Driven Ingestion: Integrate with streaming pipelines (e.g., Apache Kafka) to continuously insert embeddings from live data sources.
• Eventual Consistency vs. Strong Consistency: Systems offer trade-offs between immediate searchability of new data and system performance, allowing architects to choose the appropriate consistency model for their agent's needs.

A family of algorithms that trade a small, configurable amount of accuracy for dramatically faster retrieval speeds when searching large, high-dimensional vector datasets. Unlike exact k-Nearest Neighbors (k-NN), which performs a brute-force comparison, ANN uses intelligent indexing structures.

• Key Benefit: Enables real-time semantic search over billions of vectors.
• Common Algorithms: Hierarchical Navigable Small World (HNSW), Inverted File Index (IVF), Product Quantization (PQ).
• Trade-off: Controlled by parameters like efSearch or nprobe, balancing speed against recall.

A graph-based Approximate Nearest Neighbor (ANN) algorithm that constructs a multi-layered graph to enable extremely fast and accurate retrieval. It is a popular choice for production vector databases due to its strong performance.

• Mechanism: Creates a hierarchy of graphs, where the top layer has long-range connections for fast navigation, and lower layers have denser connections for precision.
• Advantages: High recall at low latency, relatively simple to tune.
• Implementation: Used in databases like Weaviate, Qdrant, and libraries like Faiss and hnswlib.

A retrieval strategy that combines the results of semantic (vector) search and keyword (lexical) search to improve overall recall and relevance. It mitigates the weaknesses of each approach.

• Vector Search: Excels at understanding meaning and synonyms but can be misled by typos or specific keywords.
• Keyword Search (e.g., BM25): Excels at exact term matching but fails with semantic meaning.
• Fusion Methods: Results are combined using techniques like Reciprocal Rank Fusion (RRR) or weighted scoring.

Core evaluation metrics for retrieval system performance, critical for tuning vector databases and ANN parameters.

• Recall@K: Measures the proportion of all relevant documents found within the top 'K' retrieved results. Answers: "Did we find the needle in the haystack?"
  • Formula: (Relevant items in top K) / (Total relevant items).
• Mean Reciprocal Rank (MRR): Averages the reciprocal of the rank position of the first relevant answer across multiple queries. Answers: "How high is the first good result?"
  • A perfect MRR of 1.0 means the first result was always relevant.

A two-stage retrieval process where a large set of candidate documents is first retrieved (e.g., via fast vector search), and then re-scored by a more powerful, computationally expensive model to improve final ranking precision.

• First Stage: High-recall, low-latency retrieval using a vector database.
• Second Stage: A cross-encoder model (like a fine-tuned BERT) jointly processes the query and each candidate to produce a precise relevance score.
• Benefit: Achieves high accuracy without the prohibitive cost of applying the cross-encoder to the entire corpus.

A technique that applies hard or soft constraints based on document attributes to narrow down vector search results. It is essential for building production-ready applications.

• Purpose: Combines semantic similarity with business logic or user context.
• Examples:
  • WHERE department = 'sales' AND date > '2024-01-01'
  • FILTER user_tier IN ['premium', 'enterprise']
• Implementation: Vector databases use pre-filtering (filter before search), post-filtering (filter after search), or single-stage filtered search for optimal performance.