Recursive Character Text Splitting

The algorithm operates by attempting to split text using a user-defined list of separators in a specific order of priority. Common hierarchies are:

• Primary: Double newlines (\n\n) for paragraphs.
• Secondary: Single newlines (\n) for line breaks.
• Tertiary: Sentence-ending punctuation (., !, ?) followed by a space.
• Fallback: Whitespace or character-level splitting. The splitter recursively applies the most significant separator first. If the resulting chunks are still too large, it proceeds to the next separator in the list, continuing until all chunks are within the target size range.

The core recursive loop is governed by two key parameters: chunk size and chunk overlap.

• Chunk Size: The target maximum length for a chunk, measured in characters, tokens, or other units. The algorithm's goal is to produce chunks at or below this limit.
• Recursive Application: If a split using the current separator produces a piece larger than the chunk size, that piece is fed back into the splitting function, but now using the next separator in the priority list. This continues until the piece is small enough. This ensures the final output respects the size constraint while prioritizing natural linguistic boundaries.

By prioritizing meaningful separators, this method aims to keep semantically coherent units intact for as long as possible, which is critical for retrieval quality.

• Advantage over Fixed-Length: Unlike fixed-length splitting, which can arbitrarily cut sentences in half, the recursive method will first try to split at paragraph breaks, then sentences, before resorting to arbitrary mid-sentence breaks.
• Contextual Integrity: This maximizes the likelihood that individual chunks are self-contained ideas, improving the relevance of their vector embeddings and the accuracy of semantic search.

To mitigate information loss at chunk boundaries, recursive splitters implement chunk overlap.

• Mechanism: When a split is made, a specified number of characters or tokens from the end of one chunk are duplicated at the beginning of the next chunk.
• Interaction with Recursion: Overlap is applied during the final assembly of chunks after the recursive splitting is complete. This ensures that even if a sentence is split, its context is preserved across the boundary, giving the language model a contiguous view of the text during generation.

The algorithm is defined by its list of separators, making it adaptable to different types of content.

• Code: Separators can be set to \n\n, \n, ., , ``, , for prose, or customized for specific languages (e.g., ; for C, def for Python).
• Markdown/HTML: A priority list like #, ##, \n\n, \n, . can effectively chunk by headings and paragraphs.
• Customization: Engineers can tailor the separator hierarchy to their specific corpus, making it a versatile tool beyond plain English text.

Recursive splitting is a standard utility in major LLM application frameworks.

• LangChain: The RecursiveCharacterTextSplitter class is a core document transformer. It allows configuration of separators, chunk_size, chunk_overlap, and length_function (e.g., character count vs. token count).
• LlamaIndex: Implemented via TokenTextSplitter or SentenceSplitter with a recursive mode, often abstracted within NodeParser components.
• Custom Implementations: The algorithm's simplicity makes it easy to implement from scratch, providing fine-grained control for specialized use cases not covered by libraries.

The core algorithm can be implemented directly. The pseudocode logic is:

1. Define separators in order of granularity (e.g., ["\n\n", ". ", "? ", "! ", " ", ""]).
2. Split text using the first separator in the list.
3. Check chunk size: If a resulting piece is larger than chunk_size, recursively apply the algorithm to that piece using the next separator in the list.
4. Merge small chunks with adjacent ones to avoid overly fine fragments.
5. Apply overlap by sliding a window across the final chunk list. This pattern is language-agnostic and can be optimized for specific domain documents.

Framework implementations expose key levers that engineers must tune:

• Separator Hierarchy: The order profoundly affects chunk coherence. Starting with double newlines ("\n\n") preserves paragraphs; starting with sentences (. ) creates finer chunks.
• Chunk Size vs. Overlap: A small chunk_size (e.g., 128 chars) increases retrieval precision but may fragment ideas. Overlap (e.g., 20 chars) mitigates boundary loss but increases index size and potential redundancy.
• Length Function: Using a tokenizer (like tiktoken for OpenAI models) for length_function is critical, as token counts differ from character counts, ensuring chunks fit the target model's context window.

For accurate sizing relative to an LLM's context window, recursive splitters must measure length in tokens, not characters. Frameworks allow plugging in model-specific tokenizers:

• LangChain: Use length_function=token_counter where token_counter is a function using tiktoken or transformers.
• LlamaIndex: The TokenTextSplitter is a subclass that uses token counting.
• Critical Consideration: The final chunk size must account for the prompt template tokens and the model's answer space, not just the raw text. A 512-token chunk limit often means setting the splitter's chunk_size to ~400 tokens.

A document segmentation strategy that splits text into chunks of a predetermined, uniform size, measured in characters or tokens. This method is computationally simple but often breaks sentences or ideas mid-stream.

• Primary Use: Fast, uniform preprocessing for large corpora where semantic boundaries are less critical.
• Trade-off: High risk of splitting coherent semantic units, which can degrade retrieval quality.
• Implementation: Often uses a simple sliding window with no overlap or a configurable overlap to mitigate boundary issues.

A document segmentation strategy that splits text based on natural semantic boundaries—such as paragraphs, topics, or entities—rather than arbitrary character counts. It aims to keep coherent ideas intact within a single chunk.

• Primary Use: Maximizing the contextual coherence of individual chunks for higher-quality retrieval.
• Mechanism: Relies on sentence boundary detection (SBD) and natural language understanding to identify logical breaks.
• Advantage: Produces chunks that are more meaningful to both embedding models and the final language model context.

A document segmentation strategy that creates a multi-level structure of chunks (e.g., document, section, paragraph) to enable retrieval at different levels of granularity. This supports flexible query strategies.

• Primary Use: Systems requiring both broad overview and detailed evidence retrieval from the same document.
• Structure: Often implemented with parent-child chunks, where a large 'parent' chunk contains smaller 'child' chunks.
• Retrieval: A query can first retrieve a coarse parent chunk for context, then drill down into precise child chunks for evidence.

A technique where consecutive text chunks share a portion of their content to preserve contextual continuity and mitigate information loss at chunk boundaries. It is a critical parameter in most splitting algorithms.

• Purpose: Prevents key concepts or sentences that fall on a split from being isolated, ensuring surrounding context is available in adjacent chunks.
• Implementation: Defined as a number of characters or tokens. For example, a 500-character chunk with a 50-character overlap.
• Trade-off: Increases index size and potential redundancy but is essential for maintaining retrieval recall.

The foundational process that splits raw text into smaller units called tokens, which can be words, subwords, or characters. It is a prerequisite for accurate chunking by token count and for model input.

• Importance: Language models have context limits defined in tokens, not characters. Accurate chunking requires token-aware splitting.
• Algorithms: Includes methods like Byte-Pair Encoding (BPE) used by GPT models and SentencePiece.
• Consideration: The same text can have different token counts across models, making model-specific tokenizers necessary for precise chunk sizing.

The fixed maximum sequence length of tokens that a language model can process in a single forward pass. This is the ultimate constraint that dictates the upper bound for the total size of a prompt, including retrieved chunks.

• Engineering Constraint: Defines the 'budget' for combining user queries, system instructions, and retrieved document chunks.
• Chunking Implication: The aggregate size of retrieved chunks must fit within the remaining context after accounting for the query and instructions.
• Management: Techniques like truncation or selective chunk retrieval are used when the total input exceeds this limit.