Late Chunking

The entire document is processed by a long-context embedding model before any segmentation occurs. This allows token-level embeddings to be influenced by the complete surrounding text, ensuring that a word's representation reflects its global document meaning rather than just its local sentence context. This is critical for resolving polysemy and anaphora.

After full-document encoding, the resulting token-level embeddings are segmented into chunks. Each chunk's representation is derived by pooling the contextualized token vectors within its boundaries. Common pooling strategies include:

• Mean pooling: Averages all token vectors in the chunk.
• Max pooling: Takes the maximum value per dimension.
• CLS token: Uses the embedding of a special classification token.

Because embeddings are generated before segmentation, tokens near chunk boundaries are informed by text on the other side of the split. This eliminates the boundary discontinuity problem inherent in naive chunking, where a sentence split in half loses its semantic connection. The result is a more coherent vector representation for each segment.

In a standard naive pipeline, the document is split first, and each isolated chunk is embedded independently. This causes context fragmentation: a pronoun in one chunk cannot reference its antecedent in a previous chunk. Late chunking reverses this order, embedding first and splitting second, which preserves anaphoric resolution and long-range dependencies.

Late chunking requires a long-context embedding model capable of processing an entire document in a single forward pass. This increases memory and compute requirements compared to naive chunking with a short-context model. However, the trade-off is justified when retrieval quality depends on nuanced, document-level understanding, such as in legal contract analysis or scientific literature review.

Late chunking is implemented as a preprocessing step in Retrieval-Augmented Generation systems. The workflow is:

1. Ingest the full document.
2. Generate contextualized token embeddings via a long-context model.
3. Segment the token embeddings into chunk vectors.
4. Index the chunk vectors into a vector database. This ensures that retrieved chunks carry the full weight of their original document context.

Splits documents based on semantic boundaries like paragraphs or topic shifts rather than fixed character counts. This preserves contextual integrity by ensuring that related concepts remain together within a single chunk, preventing the fragmentation of ideas that occurs with naive splitting methods.

A paradigm where each text chunk is prefixed with its document-level context before embedding. This enrichment allows the vector store to match queries against representations that maintain the original meaning, solving the problem of isolated chunks losing their surrounding narrative during retrieval.

A two-stage strategy that searches using smaller, precise child chunks to maximize relevance, then returns the larger parent chunk or full document for generation. This balances the accuracy of fine-grained matching with the completeness required for coherent language model responses.

An architecture, commonly implemented in LangChain, that indexes small chunks for search but retrieves the full parent document they originated from. This ensures the language model receives complete context, avoiding the fragmentation that occurs when only isolated chunks are provided for generation.

Establishes explicit references between related chunks, such as sequential or hierarchical connections. This enables retrieval of adjacent context or navigation through a document's structure during generation, allowing systems to reconstruct the original narrative flow from independently indexed segments.

A fine-grained method that decomposes text into atomic, self-contained propositions, each expressing a single idea. This maximizes retrieval precision for fact-checking and grounding tasks by ensuring that each indexed unit represents exactly one verifiable claim, eliminating ambiguity in matching.