Retrieval-Augmented Generation (RAG)

The Retriever is the search engine of a RAG system. It is responsible for efficiently finding the most relevant documents or text chunks from a knowledge source (e.g., a vector database) in response to a user query. Its performance is critical for the quality of the final generated answer.

• Primary Function: Maps a query to a set of candidate documents.
• Common Types: Dense retrievers (using bi-encoders like Sentence-BERT) and sparse retrievers (using lexical algorithms like BM25).
• Key Metric: Recall@K, which measures its ability to find all relevant documents within the top K results.

The Knowledge Base is the external, structured source of information the RAG system queries. It is typically a vector database (e.g., Pinecone, Weaviate, Qdrant) containing pre-processed document chunks converted into numerical embeddings.

• Ingestion Pipeline: Raw documents → Chunking → Embedding → Indexing.
• Index Type: Often uses an Approximate Nearest Neighbor (ANN) index like HNSW for fast similarity search.
• Criticality: The quality, recency, and coverage of the knowledge base directly limit the system's factual accuracy.

The Embedding Model is a neural network that converts text (queries and documents) into high-dimensional vector representations called embeddings. These embeddings capture semantic meaning, allowing the retriever to find conceptually similar text.

• Function: Creates a dense vector (e.g., 768 or 1536 dimensions) for a text input.
• Examples: OpenAI's text-embedding-ada-002, Cohere's Embed models, or open-source models like BGE-M3.
• Similarity Metric: Uses cosine similarity or dot product to measure how close a query embedding is to document embeddings in the vector space.

The Generator is the large language model (e.g., GPT-4, Claude, Llama) that produces the final natural language response. It is conditioned on both the original user query and the relevant context retrieved by the Retriever.

• Input Format: Uses a prompt template that structures the context and query. Example: "Answer based on the context: {context}\n\nQuestion: {query}"
• Core Task: Synthesizes information from the provided context to generate a coherent, accurate answer.
• Mitigates Hallucination: By grounding generation in retrieved documents, it reduces the model's tendency to invent facts.

This optional but critical component refines retrieval results for higher precision. The Query Processor may perform query expansion or rewriting. The Reranker re-scores the initial top-K retrieved documents using a more powerful, computationally intensive model.

• Reranker Model: Often a cross-encoder (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2) that jointly processes a query-document pair for a precise relevance score.
• Benefit: Dramatically improves the quality of context passed to the generator, especially when the initial retriever's recall is high but precision is moderate.

The Orchestrator is the control logic that sequences the components into a cohesive pipeline. It handles error management, latency optimization, and the flow of data from query to final answer.

• Standard Flow: Query → Embed → Retrieve → (Optionally Rerank) → Format Prompt → Generate → Return Answer.
• Advanced Patterns: Manages hybrid search (combining vector and keyword results), multi-hop retrieval, and recursive query decomposition for complex questions.
• Integration Point: This is where the RAG system is integrated into a larger application or agent framework.

Vector search is the foundational retrieval technique for RAG. It finds relevant information by comparing high-dimensional vector representations (embeddings) of a query against a pre-indexed corpus. Key aspects include:

• Similarity Metrics: Uses cosine similarity, Euclidean distance, or dot product to measure semantic closeness.
• Efficiency: For large datasets, exact search is infeasible, leading to the use of Approximate Nearest Neighbor (ANN) algorithms.
• Role in RAG: Converts the user's query into an embedding and retrieves the most semantically similar text chunks from the knowledge base to provide as context to the LLM.

Hybrid search combines multiple retrieval methods to improve the recall and precision of the context fetched for a RAG system. It typically merges:

• Semantic (Dense) Search: Uses vector embeddings to find conceptually similar content.
• Lexical (Sparse) Search: Uses keyword-matching algorithms like BM25 to find text with exact term overlap. The results from each method are fused using techniques like Recursive Rank Fusion (RRF) to produce a single, more robust ranked list. This is crucial for queries where both meaning and specific terminology are important.

Reranking is a two-stage retrieval process that refines the results from an initial, fast search. In a RAG pipeline:

1. First-Stage Retrieval: A fast method (e.g., vector search) fetches a large set of candidate documents (e.g., top 100).
2. Second-Stage Reranking: A more powerful, computationally expensive model (like a cross-encoder) re-evaluates each query-candidate pair to produce a more accurate relevance score. This improves final context quality by pushing the most relevant passages to the top before they are passed to the LLM, directly combating retrieval errors that cause hallucinations.

Approximate Nearest Neighbor (ANN) search is a family of algorithms that enable fast similarity search on massive vector datasets by trading a small amount of accuracy for massive speed gains. Essential for production RAG systems, common algorithms include:

• HNSW (Hierarchical Navigable Small World): A graph-based method known for high speed and recall.
• IVF (Inverted File Index): Clusters vectors and searches within the most promising clusters.
• PQ (Product Quantization): Compresses vectors to reduce memory footprint and accelerate distance calculations. Libraries like Faiss and Milvus provide optimized implementations of these algorithms.

Query expansion is a technique to improve retrieval recall by augmenting the user's original query. It addresses the 'vocabulary gap' where the query terms differ from the terms used in relevant documents. Methods include:

• Pseudo-Relevance Feedback: Uses top results from an initial search to extract new terms.
• LLM-Based Expansion: Uses a small language model to generate synonyms, related concepts, or a rephrased query.
• Hybrid Querying: Creates both a keyword query and an embedding from the expanded text for hybrid search. This leads to more comprehensive context retrieval, especially for short or ambiguous queries.

These are the two primary paradigms for representing text in search, each with strengths leveraged in RAG systems.

Dense Retrieval (Neural/Vector):

• Encodes queries and documents into dense, low-dimensional vectors (embeddings).
• Excels at capturing semantic meaning and handling paraphrases.
• Implemented via bi-encoder models (e.g., sentence-transformers).

Sparse Retrieval (Lexical/Keyword):

• Represents text as high-dimensional, sparse vectors (e.g., BM25, TF-IDF).
• Excels at matching specific keywords, technical terms, and named entities.
• Computationally lightweight and interpretable.

Modern RAG architectures often use a hybrid of both to balance semantic understanding and precise term matching.