Semantic Chunking

The core mechanism relies on calculating the cosine similarity between consecutive sentences or paragraphs. When the similarity score drops below a predefined threshold, a chunk boundary is inserted. This ensures that a chunk only contains text with high semantic cohesion.

• Typical Threshold: 0.5 to 0.8 cosine similarity.
• Adaptive Logic: Boundaries naturally fall at topic shifts, such as moving from a product's features to its pricing.
• Model Dependency: The quality of segmentation directly depends on the embedding model's ability to capture domain-specific nuance.

Unlike fixed-length splitting, semantic chunking prevents the fragmentation of complex ideas. By analyzing the vector representation of the text, the algorithm ensures that a logical unit—such as a full argument, a code function, or a product description—remains intact within a single chunk.

• Self-Contained Units: Each chunk represents a complete, independent idea.
• Reduced Hallucination: Providing the LLM with coherent context minimizes the risk of the model fabricating connections between unrelated facts.
• Ideal for RAG: Maximizes the signal-to-noise ratio in the retrieval pipeline.

Semantic chunking is not a one-size-fits-all process. The granularity can be dynamically adjusted by modifying the similarity threshold or by specifying a target chunk size range. A lower threshold produces larger, more general chunks, while a higher threshold creates smaller, highly specific atomic chunks.

• Coarse-Grained: Suitable for summarizing broad document themes.
• Fine-Grained: Ideal for precise fact-checking and propositional retrieval.
• Hybrid Approach: Often combined with small-to-big retrieval to search fine-grained chunks but return the larger parent context.

Because semantic boundaries align with natural document structure, they provide an ideal injection point for metadata. When a chunk is created at a topic shift, it can be automatically enriched with the preceding heading, document title, or section summary. This contextual chunking strategy significantly improves retrieval relevance.

• Structural Awareness: Links chunks to their original hierarchical position.
• Filtered Retrieval: Enables scoped searches (e.g., 'only search within the Security section').
• Improved Re-Ranking: Metadata provides strong signals for downstream re-ranking models.

To prevent information loss at the seams, semantic chunking often incorporates a chunk overlap strategy. However, unlike naive overlap, the overlap is calculated semantically to ensure that the transitional logic bridging two topics is captured in both adjacent chunks.

• Semantic Overlap: Retains the concluding sentence of a previous thought and the introductory sentence of the next.
• Pronoun Resolution: Helps resolve entities like 'it' or 'this process' that might otherwise be orphaned if the antecedent is in a different chunk.
• Graph Connectivity: Facilitates the creation of a chunk graph by maintaining explicit links between sequential segments.

Advanced semantic chunking extends beyond text to analyze other modalities. For documents containing images, tables, or charts, the algorithm can detect the surrounding descriptive text and bind the visual asset to the relevant chunk. This ensures that a table and its caption are always retrieved together.

• Table Integrity: Prevents splitting a Markdown table across two chunks.
• Image Binding: Associates vectorized image embeddings with their corresponding textual description chunks.
• AST Integration: For code, it respects Abstract Syntax Tree boundaries to keep entire functions intact.