Dense Retrieval

The embedding model (or encoder) is the neural network at the heart of dense retrieval. It maps queries and documents into a shared, low-dimensional vector space where semantic similarity corresponds to geometric proximity (e.g., cosine similarity).

• Key Types: Models are typically bi-encoders, where queries and documents are encoded independently for efficiency. Common architectures include sentence transformers like all-MiniLM-L6-v2 or fine-tuned variants of BERT.
• Training: Models are trained via contrastive learning, using positive (relevant) and negative sampling examples to pull similar items closer and push dissimilar ones apart in the vector space.

The vector index is a specialized data structure that enables fast similarity search over millions or billions of pre-computed document embeddings. A brute-force comparison is infeasible at scale.

• Algorithm Choice: For production, Approximate Nearest Neighbor (ANN) search algorithms like Hierarchical Navigable Small World (HNSW) graphs or Inverted File (IVF) indexes are used to trade minimal accuracy for orders-of-magnitude speed gains.
• Implementation: Libraries like Faiss, usearch, or commercial vector databases provide optimized implementations of these indexes, often with GPU support.

This component handles the real-time processing of user queries. The query encoder converts the incoming natural language query into a dense vector using the same embedding model that indexed the documents.

• Search Execution: The query vector is then passed to the vector index to perform a k-Nearest Neighbors (k-NN) search, retrieving the top-K most similar document vectors.
• Similarity Metric: The search uses a predefined metric, most commonly cosine similarity or inner product, to rank results. The interface returns the IDs and scores of the matched documents.

Before indexing, raw documents must be cleaned, segmented, and encoded. This offline pipeline is critical for retrieval quality.

• Chunking: Long documents are split into smaller, coherent segments (chunks) via semantic indexing and chunking algorithms to match the typical scope of a query.
• Metadata Attachment: Key attributes (source, date, author) are extracted and stored alongside each chunk's vector for metadata filtering.
• Batch Encoding: The embedding model processes all document chunks in batches to generate the persistent vector representations for the index.

A reranking model is a secondary, more powerful scorer used to refine the initial results from the fast vector index. This creates a two-stage retrieve-and-rerank pipeline for higher precision.

• Model Type: Rerankers are often cross-encoders, which jointly process the query and a candidate document, allowing for deeper interaction at the cost of higher latency. They are applied only to the top-N (e.g., 100) candidates from the first stage.
• Benefit: This hybrid approach balances the speed of bi-encoder retrieval with the accuracy of a more computationally intensive model.

This component orchestrates the entire system, handling API requests, managing the index lifecycle, and integrating with downstream applications like Retrieval-Augmented Generation (RAG).

• API Endpoints: Exposes endpoints for indexing new documents and querying the system.
• System Coordination: Manages the loading of the embedding model and the vector index, often in memory for low-latency inference.
• Observability: Includes logging for query latency, Recall@K, and other metrics to monitor performance and accuracy in production.

A bi-encoder is the neural architecture that enables dense retrieval. It uses two separate transformer models (or a shared one) to independently encode queries and documents into dense vector embeddings.

• Key Feature: Enables efficient search via pre-computed document indexes, as document embeddings can be stored and searched independently of the query.
• Contrast with Cross-Encoder: Unlike a cross-encoder that jointly processes a query-document pair for high accuracy, a bi-encoder trades some precision for massive scalability.

Approximate Nearest Neighbor (ANN) search is a family of algorithms that enable fast similarity searches over massive vector collections by trading a small, configurable amount of accuracy for orders-of-magnitude speed improvements.

• Core Problem: Exact k-NN search becomes computationally prohibitive at scale (O(n) complexity).
• Common Algorithms: Include HNSW (graph-based), IVF (clustering-based), and LSH (hashing-based).
• Use Case: The practical engine behind dense retrieval in production, allowing sub-second searches over billion-scale vector indexes.

A vector database is a specialized data management system designed for the storage, indexing, and retrieval of high-dimensional vector embeddings.

• Core Function: Provides persistent storage and optimized ANN search capabilities for embeddings generated by dense retrieval models.
• Key Features: Typically include metadata filtering, dynamic index updates, and horizontal scaling via sharding.
• Examples: Pinecone, Weaviate, Qdrant, and Milvus are dedicated vector databases. PostgreSQL with the pgvector extension adds vector capabilities to a relational DB.

Retrieval-Augmented Generation (RAG) is an architecture where a large language model's (LLM) response is grounded by first retrieving relevant context from an external knowledge source using a retriever like dense retrieval.

• Two-Stage Process: 1) Retrieval: A query is used to fetch relevant documents/chunks (often via dense retrieval). 2) Generation: The retrieved context is injected into the LLM's prompt to generate a factual, cited answer.
• Primary Benefit: Mitigates LLM hallucinations by providing an evidence base, making systems more trustworthy and updatable.

Hybrid search is a retrieval strategy that combines the strengths of multiple search methods—typically dense retrieval (semantic) and sparse retrieval (keyword-based, e.g., BM25)—into a single, more effective result set.

• Why Combine?: Dense retrieval excels at semantic understanding but can miss exact keyword matches. Sparse retrieval excels at keyword matching but fails at synonymy and paraphrasing.
• Fusion Methods: Results are combined using algorithms like Reciprocal Rank Fusion (RRF) or weighted score fusion.
• Outcome: Achieves higher recall and better overall relevance than either method alone.

Reranking is a two-stage retrieval process where a large candidate set (e.g., 100-1000 documents) is first retrieved efficiently (often via dense retrieval), then re-scored by a more powerful, computationally expensive model to improve final precision.

• Reranking Model: Typically a cross-encoder, a transformer model that jointly processes a query-document pair to produce a highly accurate relevance score. This is too slow for initial search over millions of documents.
• Architectural Role: Sits between the fast retriever (bi-encoder) and the generator (LLM) in a RAG pipeline, acting as a precision filter.
• Impact: Dramatically improves the quality of the top 5-10 results passed to the LLM.