Chunking Strategy

Chunks must represent a single, self-contained idea or topic. Breaking a document at arbitrary character counts destroys meaning.

• Structure-Aware Splitting: Parse documents by their inherent structure—headings, paragraphs, or list items—rather than fixed lengths.
• Context Preservation: A chunk should not start mid-sentence or orphan a dependent clause from its main idea.
• Example: In a legal contract, a single clause defining 'Indemnification' should be one chunk, not split across two.

Chunks must be sized to fit within the embedding model's maximum token limit while leaving room for the user query and system prompt in the final retrieval-augmented generation call.

• Token-Aware Sizing: Use the target model's tokenizer to count tokens, not words. A 'word' can be multiple tokens.
• Overhead Budgeting: Reserve a portion of the context window for the generation model's instructions and the user's question.
• Example: For a model with an 8,192 token context window, chunks of 512 tokens are common, allowing multiple chunks to be retrieved.

Introducing a controlled overlap between adjacent chunks prevents information loss at segmentation boundaries and provides essential context for isolated passages.

• Sliding Window: A small overlap (e.g., 10-15% of chunk size) ensures that concepts bridging two chunks are captured in at least one.
• Mitigating Truncation: Overlap prevents a sentence from being cut in half, where the subject is in one chunk and the predicate in another.
• Example: A 500-token chunk with a 50-token overlap ensures the last sentence of chunk A is also the first sentence of chunk B.

Each chunk must carry its provenance and structural metadata to enable precise filtering and citation during retrieval.

• Hierarchical Tagging: Attach document title, section heading, page number, and version to every chunk.
• Filtering Facets: Metadata like doc_type: "contract" or date: "2024-Q3" allows hybrid search to pre-filter candidates before semantic scoring.
• Example: A chunk from a financial report should carry metadata: {source: "10-K", section: "Risk Factors", year: 2023}.

Index smaller, highly precise chunks for accurate similarity matching, but return the larger parent context to the language model for synthesis.

• Sentence Window Retrieval: Embed individual sentences but retrieve the surrounding paragraph.
• Parent Document Retrieval: Link each small chunk to its source document block. The small chunk is used for search, the parent block is sent to the generator.
• Example: A 128-token chunk is indexed for ANN search. When retrieved, the system returns the full 1024-token section it came from.

Different document modalities require specialized chunking strategies. A one-size-fits-all approach degrades retrieval quality.

• Code: Split by functions, classes, or logical blocks using Abstract Syntax Tree (AST) parsing, not by line count.
• Tables: Preserve header context by prepending column names to each row chunk, or use a table-specific summarization model.
• Markdown: Split on ## or ### headings to maintain the natural hierarchy of the document.

The maximum number of tokens a model can process in a single forward pass. Chunking strategy is fundamentally a function of this limit. A chunk must fit within the model's context window, but also leave room for the user query, system prompt, and retrieved neighbors. Modern models like GPT-4 Turbo (128k) or Claude 3 (200k) allow for larger chunks, but longer contexts often degrade retrieval precision and increase Time to First Token (TTFT).

A buffer of tokens duplicated between adjacent chunks to prevent semantic fragmentation. If a critical sentence is split exactly at a chunk boundary, the overlap ensures it appears fully in both segments. Typical overlap values range from 10% to 20% of the chunk size. However, excessive overlap increases storage costs and can cause redundant retrieval, where the same information appears in multiple top-K results.