Chunking Strategy

Chunks must represent a complete, standalone idea. Arbitrary splits that sever sentences or paragraphs create fragments that lack the context needed for accurate semantic matching.

• Semantic Chunking uses embedding similarity to find natural break points between dissimilar sentences
• Recursive Character Text Splitting respects natural separators like paragraphs and sections
• Avoid fixed-size splitting without overlap, which can create orphaned clauses

A coherent chunk ensures the vector embedding captures the full meaning of a concept, not a truncated fragment.

Every embedding model has a maximum token limit it can process in a single forward pass. Chunks must be sized to fit within this boundary after tokenization.

• Tokenization converts raw text into model-specific numerical tokens
• A chunk that appears short in characters may exceed the limit after tokenization
• Leave headroom for the Overlap Margin and any prepended metadata like titles or section headers

Exceeding the context window causes truncation, silently discarding potentially critical information before embedding.

An Overlap Margin shares a configurable number of tokens between adjacent chunks. This prevents the loss of meaning when a sentence or concept is arbitrarily severed by a chunk boundary.

• Overlap ensures that ideas spanning two chunks are fully captured in at least one
• Typical overlap ranges from 10% to 20% of the chunk size
• Critical for dense technical documents where concepts are tightly interwoven

Without overlap, retrieval may fail because the complete definition of a term is split across two separate vectors.

Chunk size directly impacts retrieval behavior. Smaller chunks improve precision by isolating specific facts, while larger chunks provide richer context for the language model.

• Small chunks (128-256 tokens): Ideal for factoid Q&A and precise citation
• Medium chunks (512-1024 tokens): Balance context and specificity for most use cases
• Large chunks (2048+ tokens): Suitable for summarization tasks requiring broad context

The Parent-Child Chunking strategy decouples retrieval granularity from generation context, using small chunks for matching and larger parent documents for synthesis.

Document structure—headings, lists, tables—carries semantic weight. Layout Parsing and structural chunking preserve this hierarchy rather than flattening it into undifferentiated text.

• Markdown headers can be prepended to chunks as contextual metadata
• Tables should be chunked as intact units or converted to semantic text
• Code blocks require specialized splitters that respect function and class boundaries

Preserving structure ensures that a chunk containing a list item retains its relationship to the introductory sentence and surrounding items.

Every chunk should carry structural and source metadata extracted during parsing. This metadata enables filtered retrieval and provides attribution for generated answers.

• Source document title and URL for citation provenance
• Section headings to anchor the chunk within the document hierarchy
• Document type and date for authority and freshness scoring
• Chunk index and page number for precise source navigation

Metadata transforms a vector from an anonymous mathematical point into a verifiable, attributable piece of enterprise knowledge.

The maximum sequence length of tokens that an embedding or language model can process in a single forward pass, defining the hard upper boundary for chunk size.

• OpenAI text-embedding-3-large: 8,191 tokens
• Cohere embed-v3: 512 tokens per input
• Exceeding this limit triggers a truncation error or silent cropping

Chunking strategies must ensure no segment exceeds the target model's context window, or the overflowing text is lost before embedding.

An adaptive splitting technique that uses embedding similarity to determine break points between sentences, ensuring each chunk contains a self-contained, coherent idea.

• Computes cosine similarity between consecutive sentences
• Identifies break points where similarity drops below a threshold
• Produces variable-length chunks aligned with topic shifts

Unlike fixed-size splitting, semantic chunking respects the document's natural conceptual boundaries, improving retrieval precision for nuanced queries.

A hierarchical chunking method that attempts to split text using a prioritized list of separators to maintain natural semantic boundaries within a fixed chunk size.

• Separator priority: \n\n (paragraphs) → \n (lines) → . (sentences) → (words) → '' (characters)
• Recursively tries each separator until chunks fit the target size
• Balances structural integrity with token budget constraints

This is the default splitter in LangChain and LlamaIndex, offering a pragmatic middle ground between fixed-size and semantic approaches.

A configurable number of tokens shared between adjacent chunks to prevent the loss of context that occurs when a sentence is arbitrarily severed by a fixed-size splitting boundary.

• Typical overlap: 10-20% of chunk size
• Ensures key phrases spanning chunk boundaries appear in both segments
• Trade-off: increases total vector count and storage

Without overlap, a critical definition or entity mention split across two chunks becomes irretrievable by either embedding vector.

A retrieval strategy where a small, targeted child chunk is used for precise semantic matching, but the larger surrounding parent document block is returned to the language model for full context.

• Child: 256 tokens for embedding and ANN search
• Parent: 1,024+ tokens returned to the LLM for generation
• Decouples retrieval granularity from generation context

This pattern solves the tension between small chunks for accurate retrieval and large chunks for coherent answer synthesis.

The fundamental preprocessing step of segmenting raw text into atomic units called tokens, which are the numerical input IDs consumed by embedding models and language models.

• Word-level: splits on whitespace and punctuation
• Subword-level (BPE, WordPiece): breaks rare words into fragments
• Token count ≠ word count; a 500-word paragraph may be 700+ tokens

Chunking strategies must operate on token counts, not character or word counts, because the context window is measured in tokens. Miscalculating token boundaries leads to truncated embeddings.