Top-K Retrieval

K is the hyperparameter that directly controls the precision-recall trade-off. A small K (e.g., 3-5) prioritizes precision, returning only the most semantically aligned documents, which is ideal for factoid question answering. A larger K (e.g., 20-100) increases recall, providing a broader context window for downstream tasks like summarization or multi-hop reasoning. The optimal value is highly dependent on the chunking strategy and the expected density of relevant information in the corpus.

Top-K retrieval guarantees a fixed number of results, which is essential for maintaining predictable latency and context window sizes in RAG systems. In contrast, score thresholding returns a variable number of documents based on a minimum similarity cutoff. While thresholding can filter out irrelevant noise, it risks returning zero results for queries without strong matches. Production systems often combine both: applying a threshold to the Top-K candidates to ensure a minimum quality bar.

The ranking within the Top-K set is determined by a similarity function applied to the query and document embeddings:

• Cosine Similarity: Measures the angle between vectors; insensitive to magnitude.
• Dot Product: Sensitive to both angle and magnitude; often used during training with contrastive loss.
• Euclidean Distance (L2): Measures straight-line distance; requires vectors to be normalized if used for semantic similarity. The choice of metric must be consistent between model training and the vector index configuration.

Exact Top-K search over millions of high-dimensional vectors is computationally prohibitive. Top-K retrieval in production relies on Approximate Nearest Neighbor (ANN) indexes like HNSW or IVF-PQ. These structures trade a small, tunable amount of accuracy for orders-of-magnitude speedups. The ef_search parameter in HNSW, for example, directly controls the scope of the graph traversal, dynamically balancing search latency against the accuracy of the final Top-K set.

Top-K retrieval is rarely the final step. It typically serves as a first-stage retriever in a multi-stage pipeline. The initial Top-K candidates (e.g., K=100) are passed to a more computationally expensive Cross-Encoder for re-ranking. The cross-encoder performs full token-level attention between the query and each candidate, producing a highly accurate relevance score that re-orders the list before the final Top-N (e.g., N=5) are presented to the user or a language model.

The primary metric for evaluating a Top-K retriever is Recall@K: the proportion of queries for which at least one relevant document appears in the top K results. This metric is agnostic to the exact ranking position within K, making it ideal for assessing the coverage of a first-stage retriever. Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG) are used when the precise ranking order matters, such as after a re-ranking stage.

The primary evaluation metric for Top-K retrieval. It measures the proportion of all relevant documents that are successfully captured within the top-K results.

• Formula: (Relevant Documents in Top-K) / (Total Relevant Documents)
• Trade-off: Increasing K typically increases recall but may overwhelm downstream components with noise.
• Use Case: Critical for ensuring a re-ranker or LLM has access to the necessary factual grounding.

The algorithmic backbone enabling fast Top-K retrieval over billion-scale vector indexes. ANN trades a small amount of accuracy for massive speedups.

• HNSW: A graph-based algorithm building multi-layered navigable structures for logarithmic search complexity.
• IVF: Partitions the embedding space using clustering; searches only the partitions closest to the query vector.
• Goal: Find the K vectors with the highest similarity without an exhaustive linear scan.

The mathematical operation at the core of Top-K retrieval when vectors are not normalized. It finds the document vectors that maximize the dot product with the query vector.

• Relation to Cosine: If vectors are L2-normalized, MIPS is equivalent to cosine similarity.
• Asymmetric Setting: Queries and documents are often encoded by separate models, making MIPS the natural scoring function for bi-encoders.

A two-stage retrieval architecture where a fast bi-encoder performs initial Top-K retrieval, followed by a powerful cross-encoder that re-scores the candidate set.

• Bi-Encoder: Independently encodes query and document for speed.
• Cross-Encoder: Processes the concatenated query-document pair through full self-attention for precise relevance scoring.
• Pipeline: Top-K retrieval (K=1000) → Re-rank (K=100) → Final output (K=10).

Combines sparse lexical retrieval with dense semantic retrieval to generate the final Top-K list, mitigating the weaknesses of each approach.

• Sparse (BM25): Excels at exact keyword and rare term matching.
• Dense (DPR): Excels at semantic and conceptual matching.
• Reciprocal Rank Fusion (RRF): A common algorithm to merge the two ranked lists without requiring score calibration.

An evaluation metric focused on the rank position of the first relevant document. It is highly sensitive to the precision of the very top of the ranked list.

• Formula: The average of 1/rank_i across all queries, where rank_i is the position of the first correct answer.
• Contrast with Recall@K: MRR cares about the earliest hit; Recall@K cares about the total number of hits in the set.