Retrieval-Augmented Generation (RAG)

The fundamental mechanism that separates RAG from parametric memory. Instead of relying solely on internalized weights, the model's context window is dynamically populated with external documents at inference time.

• Mechanism: User query triggers a vector search; top-k results are serialized and prepended to the prompt.
• Key Distinction: This creates a non-parametric, updatable memory that can be modified without retraining the model.
• Example: A support bot answering a billing question retrieves the customer's specific invoice PDF and the latest pricing policy document, grounding its response in live data rather than training data from 6 months ago.

Production RAG systems rarely rely on a single retrieval method. Hybrid search combines the precision of sparse lexical algorithms with the semantic understanding of dense vector search to maximize recall.

• Sparse (BM25): Excels at exact keyword matching, such as product codes, legal citations, or error numbers.
• Dense (Embeddings): Captures conceptual similarity, handling synonyms and paraphrasing.
• Fusion Strategy: Reciprocal Rank Fusion (RRF) is commonly used to merge and re-rank results from both indexes, ensuring the final context list is both lexically precise and semantically relevant.

The quality of retrieval is a direct function of how source documents are segmented. Semantic chunking breaks text at natural topic boundaries rather than arbitrary token limits.

• Chunk Size: Balances specificity (small chunks) against context sufficiency (large chunks).
• Metadata Filtering: Attaching structured metadata (date, author, category) allows for pre-filtering the vector search, drastically reducing the search space.
• Overlap: A sliding window overlap prevents critical context from being severed at chunk boundaries, ensuring coherent retrieval.

A critical characteristic of trustworthy RAG is the ability to attribute every generated claim back to a source document. This is achieved through provenance tracing.

• In-Line Citations: The model is prompted to generate bracketed references [1], [2] that map to the retrieved document IDs.
• Factual Grounding: This mechanism directly mitigates hallucination by forcing the model to anchor its output in the retrieved context.
• Auditability: Provides a transparent audit trail for compliance-heavy industries like finance and healthcare, allowing a human reviewer to instantly verify the source of information.

User queries are often poorly formed for vector search. A pre-retrieval step uses an LLM to rewrite the query for optimal retrieval performance.

• HyDE (Hypothetical Document Embeddings): Generates a hypothetical ideal answer, then uses its embedding to search for similar real documents.
• Multi-Query Decomposition: Breaks a complex question into multiple sub-queries, retrieves documents for each, and synthesizes the results.
• Intent Clarification: The system identifies ambiguous queries and proactively asks for disambiguation before executing an expensive, low-confidence retrieval.

The initial vector similarity score is a weak signal of true relevance. A cross-encoder reranker is applied as a second-stage filter to reorder retrieved documents based on deep semantic interaction.

• Bi-Encoder vs. Cross-Encoder: The initial retrieval uses a fast bi-encoder; the reranker uses a slower, more powerful cross-encoder that processes the query and document jointly.
• Diversity Optimization: Prevents the final context from being dominated by near-duplicate documents, ensuring a diverse information set.
• Latency Budget: Reranking is computationally expensive, so it is typically applied only to the top-N (e.g., top-50) documents retrieved by the fast vector search.

A retrieval paradigm that uses vector embeddings to understand the contextual intent and conceptual meaning of a query rather than relying on exact keyword overlap. In RAG, semantic search identifies documents that are conceptually relevant even when they use different terminology.

• Converts queries and documents into dense vector representations
• Measures relevance via cosine similarity or dot product distance
• Contrasts with sparse lexical retrieval like BM25

A retrieval methodology that combines the precision of sparse keyword matching with the contextual relevance of dense vector similarity search. This dual approach ensures RAG systems capture both exact terminology matches and semantic relationships.

• Merges BM25 scores with embedding similarity scores
• Uses reciprocal rank fusion to combine result sets
• Prevents retrieval failures on rare terms or precise identifiers

A response synthesis strategy that strictly constrains the model's output to be derived from and supported by a provided set of source documents. This is the core mechanism that differentiates RAG from unaugmented generation.

• Requires explicit citation of source passages
• Implements attribution verification to detect unsupported claims
• Reduces hallucination by limiting the model's generative freedom

The segmentation of long-form content into discrete, self-contained semantic blocks optimized for vector database indexing and precise retrieval. Chunk size and overlap directly impact RAG retrieval quality.

• Fixed-size chunking: Splits text by token count with overlap
• Semantic chunking: Splits at natural paragraph or section boundaries
• Recursive chunking: Applies hierarchical splitting for nested documents
• Typical chunk sizes range from 256 to 1024 tokens

A set of techniques designed to prevent language models from generating factually incorrect or nonsensical information. RAG itself is a primary mitigation strategy, but additional guardrails are often layered on top.

• Factual grounding: Forcing outputs to cite retrieved evidence
• Constrained decoding: Limiting generation to valid factual paths
• Confidence calibration: Scoring output reliability against source data
• Includes post-generation factual consistency checks