Vector Search

The embedding model is the neural network that transforms raw data (text, images, audio) into dense, low-dimensional vector representations. These vectors capture the semantic meaning of the data in a continuous space.

• Function: Encodes queries and documents into a shared vector space.
• Examples: Sentence-BERT, OpenAI's text-embedding-ada-002, CLIP for multi-modal data.
• Key Property: The quality of the embeddings directly determines the upper bound of retrieval accuracy. Models are often fine-tuned on domain-specific data.

A vector index is a specialized data structure that organizes embeddings to enable fast similarity searches, avoiding the prohibitive cost of brute-force comparisons across large datasets.

• Purpose: Accelerates the Approximate Nearest Neighbor (ANN) search.
• Common Types:
  • Graph-based: Hierarchical Navigable Small World (HNSW) graphs for high recall and speed.
  • Cluster-based: Inverted File (IVF) indexes that partition space into Voronoi cells.
  • Tree-based: ANNOY (Approximate Nearest Neighbors Oh Yeah) using random projection trees.
• Trade-off: Balances between search speed, recall accuracy, and memory usage.

The similarity metric is a mathematical function that quantifies the closeness or distance between two vectors in the embedding space, determining the ranking of search results.

• Cosine Similarity: Measures the cosine of the angle between vectors. Most common for semantic search as it is magnitude-invariant.
• Euclidean Distance (L2): Measures the straight-line distance between points in the vector space.
• Inner Product (Dot Product): Used for Maximum Inner Product Search (MIPS), crucial in recommendation systems.
• The choice of metric must align with the embedding model's training objective for correct results.

This component manages the retrieval pipeline, often implementing multi-stage search strategies to balance speed and precision.

• Query Planner: Executes the search strategy, which may involve:
  • Hybrid Search: Combining vector and sparse (e.g., BM25) keyword search results using fusion methods like Reciprocal Rank Fusion (RRF).
  • Metadata Filtering: Applying hard filters (e.g., date, category) before or after the vector search.
• Reranker: A second-stage model (often a cross-encoder) that re-scores a small set of candidate documents (e.g., Top-K results) for higher precision, at greater computational cost.

The vector database is the persistence and serving layer that houses the vector index, original data, and associated metadata. It provides the APIs for CRUD operations and search.

• Core Functions:
  • Storage: Persists vectors, metadata, and raw content.
  • Index Management: Handles index creation, updates, and versioning.
  • Query Serving: Exposes endpoints for similarity search with filtering.
• Scalability Features: Often includes sharded indexes for horizontal scaling and in-memory caching for low-latency queries.
• Examples: Pinecone, Weaviate, Qdrant, Milvus.

Evaluation metrics quantitatively measure the performance of a vector search system, guiding optimization and benchmarking against requirements.

• Recall@K: The proportion of all relevant documents found within the top K retrieved results. Measures completeness.
• Mean Reciprocal Rank (MRR): Averages the reciprocal rank of the first relevant answer across multiple queries. Measures how high the first relevant result ranks.
• Latency: Query response time, typically measured in milliseconds at specific percentiles (p95, p99).
• Throughput: Queries per second (QPS) the system can handle under load.
• These metrics create the trade-off curve between accuracy, speed, and cost.

A family of algorithms that trade a small, configurable amount of accuracy for significantly faster retrieval speeds when searching massive, high-dimensional vector datasets. Unlike exact k-NN, ANN uses techniques like graph traversal or quantization to avoid comparing the query to every vector in the database.

• Key Algorithms: Include HNSW, IVF, and Product Quantization.
• Trade-off: Controlled by parameters like ef (HNSW) or nprobe (IVF) which balance speed against recall.
• Use Case: Essential for production-scale vector search where query latency is critical, such as real-time recommendation or RAG systems.

A graph-based approximate nearest neighbor search algorithm that constructs a multi-layered graph to enable extremely fast and accurate retrieval. It is a leading algorithm for in-memory vector search due to its high recall and low latency.

• Mechanism: Creates a hierarchical graph where long-distance links on top layers enable fast traversal, and dense connections on lower layers provide high accuracy.
• Performance: Often achieves sub-millisecond query times on million-scale datasets with high recall (>0.95).
• Implementation: Found in libraries like Faiss, Weaviate, and Qdrant as a primary indexing method.

The most common metric for measuring semantic similarity between vector embeddings in high-dimensional spaces. It calculates the cosine of the angle between two vectors, making it magnitude-invariant.

• Formula: cos(θ) = (A · B) / (||A|| ||B||)
• Range: Outputs a value between -1 (perfectly opposite) and 1 (identical direction). For normalized vectors, this simplifies to a dot product.
• Application: The default similarity metric for most embedding models (e.g., OpenAI's text-embedding-ada-002) and vector databases. It is preferred over Euclidean distance for semantic similarity as it focuses on orientation, not magnitude.

A retrieval strategy that combines the results of semantic (vector) search and keyword-based (lexical) search to improve overall recall and relevance. It addresses the weaknesses of each method: vector search's potential for missing exact keyword matches and keyword search's inability to understand semantics.

• Implementation: Typically involves running both a vector search and a BM25 search, then fusing the ranked result lists using algorithms like Reciprocal Rank Fusion (RRF).
• Benefit: Provides high recall by retrieving documents that are semantically similar and those containing exact query terms.
• Example: Searching for "Python" should return documents about the programming language (semantic) and those specifically mentioning the word "Python" (lexical), while filtering out those about the snake.

A neural search paradigm where queries and documents are encoded into dense, low-dimensional (e.g., 384 or 768-dimension) vector embeddings using a transformer-based model (a bi-encoder). Relevance is determined by the similarity (e.g., cosine) between these dense vectors.

• Contrast with Sparse Retrieval: Uses dense, continuous vectors instead of sparse, high-dimensional bag-of-words vectors (TF-IDF, BM25).
• Training: Models like DPR are fine-tuned on (question, positive passage, negative passage) triples using contrastive loss.
• Advantage: Captures semantic meaning and synonymy, enabling searches for concepts not explicitly mentioned in the text.

A two-stage retrieval process designed to improve precision. A fast, initial retrieval method (e.g., vector search) fetches a broad set of candidate documents (e.g., top 100). A more powerful, computationally expensive cross-encoder model then re-scores this smaller set.

• Cross-Encoder: A transformer model that jointly processes the query and a single document together, allowing for deep, attention-based interaction. This produces a highly accurate relevance score but is too slow to run against an entire corpus.
• Efficiency Trade-off: Achieves near the quality of using a cross-encoder on the full database at a fraction of the computational cost.
• Outcome: The final top-K results (e.g., top 3) are much more likely to be precisely relevant to the query.