Semantic Chunking

Splits documents at natural semantic boundaries—paragraphs, sections, or topic shifts—rather than fixed token counts. This preserves the contextual integrity of each chunk, preventing ideas from being severed mid-thought. Common implementations use sentence embeddings to detect when cosine similarity between consecutive sentences drops below a threshold, signaling a topic transition. This approach ensures that a chunk about 'transformer attention mechanisms' isn't arbitrarily fused with unrelated content about 'training data preprocessing.'

The core mechanism relies on computing pairwise cosine similarity between adjacent sentences or paragraphs using a sentence-transformer model. When similarity falls below a configurable percentile—typically the 90th percentile of all adjacent similarities—a chunk boundary is inserted. This adaptive approach means dense, tightly-argued sections produce larger chunks, while discursive content with frequent topic shifts generates smaller ones. The threshold is tunable: lower percentiles yield more granular chunks for precise retrieval; higher percentiles produce larger, more context-rich chunks.

To mitigate information loss at boundaries, semantic chunking often employs overlapping windows where adjacent chunks share a configurable number of sentences. This redundancy ensures that a concept spanning a boundary is fully retrievable from either chunk. A typical configuration retains 10-20% overlap between consecutive chunks. More sophisticated implementations use sliding window approaches where the overlap region is dynamically sized based on the semantic cohesion of the boundary zone, maximizing both recall and storage efficiency.

Documents are often segmented into a multi-level hierarchy reflecting their inherent structure:

• Level 1 (Document): Full text with metadata
• Level 2 (Section): Major headings and their content
• Level 3 (Paragraph): Individual semantic blocks
• Level 4 (Sentence): Atomic units for fine-grained retrieval

This enables small-to-big retrieval, where a query matches a small child chunk but the system returns the larger parent context for generation, balancing precision with completeness.

Each chunk is annotated with structural and provenance metadata during segmentation, including:

• Section headings and their hierarchy
• Positional index within the source document
• Document title, author, and publication date
• Preceding and following chunk IDs for context expansion

This metadata enables filtered retrieval—e.g., searching only chunks from documents published in the last 90 days—and supports chunk linking for retrieving adjacent context when a matched chunk requires surrounding information for coherent generation.

Fixed-size chunking (e.g., 512 tokens with 10% overlap) is computationally simple but frequently fragments concepts mid-thought, degrading retrieval accuracy. Semantic chunking produces variable-length chunks that respect document structure, leading to:

• Higher retrieval precision: Chunks contain complete ideas, improving vector matching
• Reduced hallucination: Generation models receive coherent context, not fragmented snippets
• Better citation fidelity: Source attribution maps cleanly to logical document units

The trade-off is increased preprocessing complexity and the need for an embedding model during the chunking pipeline itself.

A paradigm where each text chunk is prefixed with its document-level context before embedding. This enriches the vector representation, allowing the store to match queries against chunks that retain their original meaning rather than isolated fragments.

• Prevents loss of meaning from decontextualized snippets
• Implemented by prepending document titles and summaries
• Critical for long-form technical documentation

A technique where a long document is embedded in its entirety first using a long-context model, and the resulting token-level embeddings are then segmented into chunks. This preserves cross-chunk contextual awareness that is lost when chunking before embedding.

• Contrasts with the standard 'embed chunks' pipeline
• Leverages models with extended context windows
• Maintains coherence across artificial boundaries

A strategy that searches using smaller, precise child chunks to maximize relevance, then returns the larger parent chunk or full document for generation. This balances the accuracy of fine-grained search with the completeness required for coherent synthesis.

• Child chunks: 100-200 tokens for precise matching
• Parent chunks: 500-1000+ tokens for generation context
• Prevents the 'lost context' problem of small chunks

A fine-grained method that decomposes text into atomic, self-contained propositions—each expressing a single fact. This maximizes retrieval precision for fact-checking and grounding tasks by ensuring every indexed unit is a complete, independent statement.

• Uses an LLM to split complex sentences
• Each chunk is a single verifiable claim
• Ideal for high-precision Q&A and contradiction detection

Combines dense vector similarity with sparse keyword retrieval like BM25. This leverages semantic understanding for conceptual matches while preserving exact term matching for names, codes, and rare vocabulary.

• Dense: captures meaning and paraphrases
• Sparse: excels at exact identifiers and keywords
• Often fused via Reciprocal Rank Fusion (RRF)

A two-stage refinement where a computationally expensive cross-encoder jointly processes the query and each candidate chunk to produce a precise relevance score. Applied only to the top-k results from a faster initial retrieval to balance cost and accuracy.

• Stage 1: Fast bi-encoder or ANN search
• Stage 2: Precise cross-encoder scoring
• Dramatically improves final result quality