Chunk Overlap

Chunk overlap functions as a sliding window across the source document. When a chunk ends at token N, the next chunk begins at token N - overlap_size rather than N + 1. This creates a redundant buffer where the tail of one chunk becomes the head of the next.

• Example: With a chunk size of 512 tokens and an overlap of 64 tokens, chunk 2 starts at token 448 of the original sequence.
• Trade-off: Higher overlap increases storage costs and indexing time but reduces the risk of splitting critical context.
• Typical range: 10% to 20% of the chunk size is a common starting heuristic.

Without overlap, a sentence or logical unit can be arbitrarily severed at the chunk boundary. The embedding of a truncated sentence loses semantic fidelity because it lacks the surrounding syntactic and conceptual cues.

• Problem: A definition split across two chunks becomes two incomplete vectors, neither of which can answer the query independently.
• Solution: Overlap ensures that the complete sentence appears intact in at least one chunk.
• Impact: Improves recall for queries whose answers sit near segmentation points.

Overlap is not an independent variable; it is a function of chunk size and document structure. A fixed overlap of 50 tokens behaves very differently on a 256-token chunk versus a 1024-token chunk.

• Small chunks (128-256 tokens): High relative overlap (20-25%) is often necessary because the window is too small to capture full paragraphs.
• Large chunks (1024+ tokens): Lower relative overlap (5-10%) may suffice, as the chunk itself already contains substantial context.
• Dynamic adjustment: Advanced pipelines vary overlap based on the semantic density of the boundary region.

While overlap improves recall, it introduces a precision trade-off. Redundant content across chunks means multiple vectors may match the same query, diluting the diversity of retrieved results.

• Duplicate context: The LLM may receive the same sentence from two different chunks, wasting precious context window tokens.
• Mitigation strategies:
  • Apply Maximal Marginal Relevance (MMR) during retrieval to penalize redundant chunks.
  • Use chunk deduplication post-retrieval to collapse overlapping segments.
  • Tune overlap downward if retrieval consistently returns near-identical chunks.

The role of overlap differs fundamentally between fixed-length and semantic chunking strategies.

• Fixed-length chunking: Overlap is a blunt instrument—the only mechanism to prevent mid-word or mid-sentence breaks. It is mathematically predictable but semantically blind.
• Semantic chunking: Splitting occurs at natural topic boundaries identified by embedding similarity drops. Overlap here acts as a safety net for edge cases where a boundary detector fails.
• Hybrid approach: Some systems apply a minimal fixed overlap even on semantic boundaries to guarantee continuity across imperfect splits.

Overlap granularity can be defined at the token, word, or sentence level, each with distinct behaviors.

• Token-level overlap: The most granular; ensures no token pair is lost. Common in fixed-length splitters using tiktoken or similar tokenizers.
• Sentence-level overlap: Overlaps by a fixed number of complete sentences. Preserves syntactic integrity but may overshoot or undershoot token budgets.
• Paragraph-level overlap: Used in parent document retrieval where small child chunks overlap within a larger parent context.
• Best practice: Align overlap granularity with the natural delimiter of your content type (sentences for prose, lines for code).