Semantic Chunking

The core mechanism relies on computing the cosine similarity between consecutive text segments. When the similarity score drops below a predefined threshold, a new chunk boundary is inserted. This ensures that chunks represent coherent, topically unified units rather than arbitrary text slices.

• Common thresholds: 0.5 to 0.8 cosine similarity
• Dynamic boundaries: Chunk sizes vary based on content, not a fixed token count
• Implementation: Uses sentence transformers or OpenAI embeddings to vectorize text

Documents are split using a recursive, top-down approach that respects natural structural separators. The algorithm first attempts to split on document-level boundaries (headings, sections), then paragraph breaks, then sentence boundaries, and finally word boundaries if no other separator is found.

• Preserves the logical hierarchy of the source document
• Prevents orphaned sentences or mid-thought breaks
• Maintains contextual integrity for downstream retrieval

To prevent information loss at chunk boundaries, semantic chunking often incorporates a configurable overlap between adjacent chunks. This overlap acts as a contextual bridge, ensuring that concepts spanning a boundary are not fragmented.

• Typical overlap: 10-20% of chunk size
• Prevents the Lost-in-the-Middle retrieval failure mode
• Critical for RAG systems where a single fact may be split across a naive boundary

Each semantically chunked segment is enriched with structural metadata that preserves its position within the original document hierarchy. This includes section titles, heading paths, page numbers, and document identifiers.

• Enables source attribution in generative outputs
• Supports filtered retrieval (e.g., 'only search within Section 3')
• Facilitates re-ranking by document structure relevance

Semantic chunking dynamically adapts chunk sizes to fit within the context window constraints of target embedding models and LLMs. Unlike fixed-size chunking, it balances semantic completeness with token limits.

• Embedding model limits: e.g., 512 tokens for text-embedding-ada-002
• LLM context budgets: Chunks sized to leave room for prompt, history, and generation
• Avoids truncating critical information mid-concept

Semantic chunking is the critical pre-processing step before vector database ingestion. The quality of chunks directly determines retrieval precision and recall in RAG architectures.

• Chunks are embedded and stored in vector stores like Pinecone, Weaviate, or pgvector
• Poor chunking leads to hallucination-prone retrieval
• Often paired with Hybrid Search (dense + sparse) for optimal recall

The maximum span of text, measured in tokens, that an LLM can process in a single inference request. This defines the upper limit of the model's immediate working memory. Modern models range from 4K tokens in legacy systems to 128K+ in frontier models like GPT-4 Turbo and Claude 3. Exceeding this limit triggers truncation, where content is silently discarded—often from the middle of the document—leading to catastrophic retrieval failures.

A documented performance failure mode where information placed in the center of a long context window is significantly less likely to be retrieved or utilized compared to content at the beginning or end. This U-shaped attention curve means that even when a relevant chunk is technically within the context window, the model may functionally ignore it. Mitigation strategies include re-ranking to push critical chunks to the top and bottom, and explicit position-aware prompting.

A second-pass retrieval stage where a more computationally intensive, high-precision model re-scores an initial set of candidate documents. The goal is to ensure the most semantically relevant chunks are placed at the privileged top positions of the context window. Common approaches include cross-encoder models like Cohere Rerank or BGE-Reranker, which evaluate query-document pairs jointly rather than relying solely on vector similarity.

A retrieval strategy that combines the semantic understanding of dense vector search with the precise keyword matching of sparse lexical search (e.g., BM25). Results are typically fused using Reciprocal Rank Fusion (RRF). This dual approach ensures that both conceptual matches and exact terminology matches are captured, which is essential when semantic chunks contain domain-specific jargon that embedding models may not fully capture.

The process of segmenting a raw text string into a sequence of discrete atomic units called tokens from a fixed vocabulary. Tokenization directly impacts semantic chunking because chunk boundaries defined by token count may split words or sentences mid-stream if not aligned with the tokenizer's behavior. Subword methods like Byte-Pair Encoding (BPE) mean that a 512-token chunk is not equivalent to 512 words—it's typically 350–400 words depending on domain vocabulary.