Retrieval-Augmented Generation (RAG)

Modern RAG systems rarely rely on a single retrieval method. Hybrid Search combines the semantic understanding of dense vector embeddings with the precise keyword matching of sparse lexical algorithms like BM25. This fusion, typically orchestrated via Reciprocal Rank Fusion (RRF), ensures that both conceptual relevance and exact term matches are captured, dramatically improving recall for queries containing specific identifiers, codes, or rare entities that dense models might overlook.

The atomic unit of retrieval is the chunk. Semantic Chunking segments documents based on meaning—using paragraph breaks, topic shifts detected by embedding similarity, or section headers—rather than arbitrary character counts. This prevents the 'semantic truncation' problem where a critical concept is split across two chunks, rendering both useless. Effective chunking preserves self-contained context, ensuring each retrieved piece is a complete, coherent thought that the LLM can process without prior context.

Retrieved documents often contain extraneous text that wastes precious token space. Contextual Compression uses a secondary, lightweight model to extract only the minimal sentences or facts relevant to the user's query from each retrieved chunk. This process maximizes the information density of the prompt, allowing the generator to consider a broader set of sources within the fixed context window without being distracted by noise.

The core value proposition of RAG is grounding generation in verifiable data to mitigate hallucination. This requires explicit citation signal engineering. The prompt instructs the LLM to attribute every factual claim to a specific source chunk ID. Post-generation, these citations are parsed to create hyperlinks. This establishes a verifiable chain of provenance, transforming the LLM from an opaque oracle into an auditable reasoning engine that can be fact-checked against its sources.

In production RAG systems, the system prompt and retrieved context form a static prefix for many user queries. Prefix Caching stores the computed KV-Cache for this shared prefix in GPU memory. When a new query arrives with the same system prompt and retrieved documents, the prefill phase is skipped entirely, slashing Time to First Token (TTFT) and reducing redundant computation. This is critical for scaling RAG to high-throughput, low-latency applications.

A content segmentation strategy that divides a document into chunks based on semantic boundaries—such as paragraph breaks or topic shifts detected by embedding similarity—rather than a fixed character or token count. This ensures that each chunk represents a coherent unit of meaning, preventing the fragmentation of concepts across retrieval units. Effective chunking directly impacts the precision of the retrieval step in a RAG pipeline by ensuring the retriever can match queries to self-contained, relevant passages.

A retrieval strategy that combines the semantic understanding of dense vector search with the precise keyword matching of sparse lexical search (like BM25). Results are typically fused using Reciprocal Rank Fusion (RRF) to improve recall. In a RAG architecture, hybrid search mitigates the failure modes of relying solely on embeddings, ensuring that documents containing exact terminology or rare entities are not missed by the semantic retriever.

A second-pass retrieval stage where a more computationally intensive, high-precision model re-scores an initial set of candidate documents. The goal is to ensure the most semantically relevant chunks are placed at the top of the context window before generation. This is critical in RAG because the language model's attention mechanism is biased toward the beginning and end of the prompt; re-ranking maximizes the chance that the most salient information occupies these high-attention positions.

A retrieval augmentation technique where a secondary model compresses retrieved documents by extracting only the minimal information necessary to answer the user's query. This maximizes the utility of the limited context window by filtering out extraneous text. In a RAG pipeline, contextual compression prevents the prompt from being bloated with irrelevant passages, reducing both latency and the risk of the model being distracted by noise.

A documented performance failure mode of LLMs where information placed in the center of a long context window is significantly less likely to be retrieved or utilized compared to information at the beginning or end. This phenomenon has direct architectural implications for RAG systems: even if the retriever fetches the correct document, its placement in the assembled prompt can determine whether the model effectively grounds its generation on that evidence.

An encoder-decoder architecture for knowledge-intensive tasks that processes each retrieved document passage independently in the encoder and performs joint reasoning over all passages in the decoder through cross-attention. Unlike standard RAG approaches that concatenate all documents into a single prompt, FiD scales to hundreds of retrieved passages by isolating encoding, allowing the decoder to synthesize information from a much larger retrieval set.