Markdown Header Splitting

The algorithm parses a document's Abstract Syntax Tree (AST) to identify header nodes (# Heading 1, ## Heading 2). It creates a chunk boundary at each header, ensuring the resulting segment contains all content from that header until the next header of equal or higher rank. This preserves the author's intended document hierarchy and logical flow, making chunks inherently semantically coherent. For example, a ## Methods section and all its sub-sections (### Data Collection, ### Analysis) would form a single, logically unified chunk.

Splitting is governed by header levels, not fixed character counts. Engineers configure a split depth (e.g., split at ## but not ###). Key parameters include:

• Header Level to Split On: Defines the primary chunk boundary (e.g., ## for H2).
• Max Chunk Size: A safety fallback; if a section exceeds this, recursive splitting using lower-level headers or sentence boundaries is triggered.
• Chunk Overlap: A configurable number of characters or sentences can be carried over from the previous chunk to prevent context loss at boundaries, crucial for embedding generation.

Chunks created by header splitting are optimal for vector embedding and retrieval because they align with topical units. This improves retrieval precision in RAG systems, as queries are matched against whole, self-contained sections rather than arbitrary text spans. The technique reduces topic fragmentation, where a single concept is split across multiple chunks, and topic conflation, where multiple concepts are merged into one noisy chunk. This leads to higher embedding fidelity and more relevant context being injected into the LLM's prompt.

This logic is implemented in popular libraries. Key implementations include:

• LangChain's MarkdownHeaderTextSplitter: Takes a list of headers to split on (e.g., ["#", "##", "###"]) and returns chunks with metadata specifying the source headers.
• LlamaIndex's MarkdownNodeParser: Creates DocumentNode objects where each node's metadata contains the header path (e.g., ['# Overview', '## Key Features']).
• Custom Parsers: Often built using Python's markdown library to generate an AST, followed by a tree-walking algorithm to extract sections.

Markdown header splitting is a rule-based, content-aware method, distinct from other common techniques:

• Vs. Recursive Character Splitting: More semantically intelligent than splitting by paragraphs/sentences alone, as it uses document structure.
• Vs. Semantic Chunking: Does not require computing embeddings to find boundaries, making it faster and deterministic. However, it is less flexible for unstructured text.
• Vs. Fixed-Size Chunking: Avoids decapitating headers from their content, a common failure mode of naive character-window approaches. It is often used in a hybrid pipeline, where header splitting creates primary chunks, and recursive splitting sub-divides any that exceed a size limit.

While powerful, the technique has specific constraints engineers must address:

• Markdown-Dependency: Only works on documents with proper header syntax. Poorly formatted docs require preprocessing.
• Nested Complexity: Deeply nested documents (e.g., H5) can create very small or imbalanced chunks.
• Metadata Propagation: Crucial to preserve header titles as metadata for filtered retrieval. For example, allowing a search to be scoped to only ## API Reference sections.
• Headerless Content: Introductory or concluding text before the first header requires a separate handling strategy, often a prefix chunk.

Semantic chunking segments a document into coherent units based on contextual meaning and topic boundaries, rather than arbitrary character or token counts. This method aims to keep semantically related information together to maximize the relevance of retrieved chunks.

• Core Principle: Split at natural topic shifts identified through lexical analysis or embedding similarity.
• Contrast with Header Splitting: While header splitting uses explicit structural markers, semantic chunking infers boundaries from the text's content itself.
• Use Case: Ideal for unstructured text without clear formatting, or to create more nuanced segments than headers alone provide.

Recursive character text splitting is a widely used algorithm that recursively divides text using a hierarchy of separators until chunks of a desired size are created.

• Mechanism: It first attempts to split by larger separators (e.g., \n\n for paragraphs), then recursively by smaller ones (e.g., \n, . , ) if chunks remain too large.
• Key Feature: Balances adherence to natural boundaries (like sentences) with strict size constraints.
• Practical Application: The default chunking method in many RAG frameworks and libraries like LangChain, providing a robust baseline for diverse document types.

Sentence boundary detection (SBD) is the NLP task of identifying the start and end points of sentences within a body of text. It is a foundational preprocessing step for higher-level semantic chunking.

• Challenge: Abbreviations like "Dr." or decimal points can falsely signal sentence ends.
• Methods: Ranges from rule-based systems using punctuation and capitalization to machine learning models trained on annotated corpora.
• Downstream Impact: Accurate SBD is critical for creating high-quality sentence or paragraph-level embeddings used in semantic search and chunking algorithms.

Embedding-based chunking uses sentence or paragraph embeddings to measure semantic similarity and identify natural topic shifts within a document.

• Process: Computes embeddings for sequential text units (e.g., every 2-3 sentences). Chunk boundaries are placed where the cosine similarity between consecutive units drops below a threshold, indicating a topic change.
• Advantage: Creates chunks that are internally cohesive in meaning, which can improve retrieval precision.
• Tooling: Relies on models like Sentence-BERT (SBERT) to generate the dense vector representations used for similarity comparison.

The TextTiling algorithm is a classic, unsupervised method for segmenting text into multi-paragraph topical units by analyzing patterns of lexical cohesion.

• Core Idea: It slides a fixed-width window across the text, calculating a "depth score" based on term co-occurrence. Valleys in the resulting score plot indicate potential topic boundaries.
• Characteristic: Operates without pre-training or understanding of grammar, relying solely on term frequency patterns.
• Legacy: Serves as a historical precursor to modern embedding-based methods and is effective for segmenting long, structured expository text.

A sliding window chunk is created by moving a fixed-size context window across text with a specified overlap between consecutive chunks.

• Primary Purpose: To preserve context across arbitrary split points and mitigate information loss at chunk boundaries. The overlap ensures that concepts spanning a split are still captured.
• Trade-off: Increases the total number of chunks and potential retrieval redundancy but is simple and guarantees consistent chunk sizes.
• Typical Configuration: A window of 500 tokens with a 50-token overlap is a common starting point for RAG pipelines processing varied documents.