Semantic Chunking

This algorithm uses a hierarchy of separators (e.g., \n\n, \n, . , ) to recursively split text until chunks are within a target size range. It prioritizes larger semantic units first, like paragraphs, before breaking them down further.

• Primary Use: General-purpose chunking where document structure is varied or unknown.
• Key Benefit: Balances semantic coherence with strict size constraints for fixed-context windows.
• Implementation: Libraries like LangChain and LlamaIndex provide configurable recursive splitters.

This technique identifies natural topic shifts within text to create chunks where internal content is maximally cohesive. It often uses sentence embeddings to measure similarity between adjacent blocks of text.

• Core Mechanism: Calculates cosine similarity between embedding vectors of sequential sentences or paragraphs. A significant drop indicates a likely topic boundary.
• Advantage: Produces chunks that align with the document's conceptual flow, not arbitrary length.
• Tool Example: The semantic-text-splitter Python library implements this approach.

This method leverages explicit document structure and markup to guide segmentation. It respects the author's intended organization.

• Markdown/HTML Header Splitting: Chunks are created at heading tags (<h1>, #, ##), producing sections that mirror the document outline.
• Code Syntax Splitting: For technical documentation, splits can occur at function/class definitions or logical code blocks.
• Benefit: Preserves high-level semantic context and hierarchical relationships, which is critical for accurate retrieval.

A fixed-size window moves across the text, creating chunks with a specified character or token overlap between consecutive segments.

• Primary Use: Mitigating context fragmentation, where a key piece of information is cut in half at a chunk boundary.
• How it Works: A 1000-token window with a 200-token overlap ensures 200 tokens of context are repeated in the next chunk, preserving continuity.
• Trade-off: Increases total number of chunks and potential retrieval redundancy, which must be managed by downstream ranking.

This strategy uses Named Entity Recognition (NER) to inform split decisions, aiming to keep all mentions of a specific entity (person, organization, location) within a single chunk.

• Objective: Preserve the full contextual narrative around an entity for downstream tasks like question answering or relationship extraction.
• Process: The chunker analyzes entity density and proximity; it will avoid splitting text between closely clustered mentions of the same entity.
• Result: Improves the likelihood that a retrieved chunk contains complete information about a queried subject.

Classical NLP algorithms designed specifically for detecting topical boundaries in unstructured text.

• TextTiling Algorithm: An unsupervised method that analyzes patterns of lexical cohesion (term co-occurrence) across a moving window. It identifies 'valleys' in similarity scores as segment boundaries.
• C99 Algorithm: A linear segmentation algorithm that uses a rank matrix of sentences to identify coherent segments.
• Use Case: Effective for segmenting long-form narratives, transcripts, or articles where explicit structural markers are absent.

A foundational chunking algorithm that uses a hierarchy of separators (e.g., \n\n, ., ,, ) to recursively split text until chunks fall within a target size range. It prioritizes keeping paragraphs and sentences intact but will break them if necessary to meet size constraints.

• Key Feature: Balances semantic coherence with strict size limits.
• Use Case: A reliable, language-agnostic fallback when more sophisticated semantic parsing is unavailable or too costly.

The NLP task of identifying where sentences begin and end within unstructured text. It is a critical preprocessing step for high-quality semantic chunking, as sentences are natural semantic units.

• Challenge: Ambiguities like periods in abbreviations (e.g., Dr. Smith) or decimal points.
• Tools: Libraries like spaCy, NLTK, and modern transformer-based models provide robust, multilingual sentence segmentation.

A segmentation method that uses sentence or paragraph embeddings to measure semantic similarity across a document. It identifies natural topic boundaries by detecting significant drops in cosine similarity between consecutive text segments.

• Process: Embeds sliding windows of text, calculates similarity scores, and splits where scores dip below a threshold.
• Advantage: Creates chunks that are internally cohesive, often aligning with human perception of topic shifts.

An unsupervised, lexical cohesion-based algorithm for segmenting text into multi-paragraph topical units. It works by analyzing term frequency patterns within a moving window.

• Mechanism: Computes a cohesion score based on term co-occurrence between adjacent blocks of text. Valleys in the score plot indicate likely topic boundaries.
• Characteristic: Does not require pre-trained models, making it fast and applicable to domain-specific jargon.

A retrieval strategy that combines the results of sparse (keyword-based, e.g., BM25) and dense (vector similarity) search methods. The effectiveness of semantic chunking is directly tested in hybrid search systems.

• Rationale: Keyword search excels at exact term matching; vector search captures semantic meaning. Together, they cover a wider range of query types.
• Implementation: Typically involves fetching candidate sets from both indexes and using a weighted score (e.g., α * BM25_score + (1-α) * similarity_score) to rank the final results.

A graph-based algorithm for performing fast approximate nearest neighbor (ANN) search in high-dimensional spaces. It is the most common indexing method for the vector embeddings generated from semantically chunked text.

• Performance: Provides an excellent trade-off between recall, speed, and memory usage compared to other ANN methods.
• Ubiquity: The default or highly optimized index in major vector databases like Weaviate, Qdrant, and Milvus, and libraries like FAISS.