Chunk Granularity

Fine-grained chunking splits text into its smallest coherent units, such as individual sentences or short phrases. This approach maximizes retrieval precision by allowing the system to pinpoint the exact sentence containing an answer.

• Best For: Factoid questions, direct quotations, and queries requiring high specificity.
• Trade-Off: Can suffer from poor recall if the answer requires broader context, and may increase computational overhead due to a larger number of chunks to index and search.
• Example: Chunking a research paper into individual sentences to find the exact statement of a hypothesis.

Medium-grained chunking uses natural semantic boundaries like paragraphs, subsections, or topics. This balances context preservation with retrievability.

• Best For: Most general-purpose RAG applications. Provides enough surrounding context for the LLM to interpret the retrieved information without being overwhelmed.
• Implementation: Often achieved via semantic chunking or recursive splitting using separators like \n\n for paragraphs.
• Example: Splitting a product manual so each chunk contains a procedure, its prerequisites, and expected outcomes.

Coarse-grained chunking creates large segments, such as entire document sections or full articles. This prioritizes contextual completeness and recall for complex, multi-faceted queries.

• Best For: Summarization tasks, queries requiring synthesis across multiple concepts, or when using LLMs with very large context windows.
• Trade-Off: Can severely degrade precision, retrieving large blocks of irrelevant text and wasting precious context window tokens.
• Example: Using an entire chapter of a legal statute as a single chunk to ensure all interrelated clauses are presented together.

Granularity creates a fundamental engineering trade-off between precision and recall.

• High Precision (Fine): Retrieves exactly what is needed but may miss relevant information split across chunks or requiring context.
• High Recall (Coarse): Retrieves all potentially relevant information but includes more noise, forcing the LLM to filter.

The optimal point on this curve is determined by the query domain, the LLM's context window size, and the required answer quality.

Advanced systems bypass the single-granularity limitation by using multi-level strategies.

• Hierarchical Chunking: Creates a tree of chunks (e.g., document > section > paragraph). A query can first retrieve a coarse parent chunk, then drill down into relevant fine-grained child chunks.
• Parent-Child Chunks: Enables a two-stage retrieval where a small, dense embedding for a child chunk is used for a fast, precise search, and its larger parent chunk provides full context for generation.
• Sentence Window Retrieval: A hybrid where a single sentence (fine) is embedded and retrieved, and a fixed window of surrounding sentences (medium) is appended for context.

Several technical constraints directly influence the choice of granularity.

• Model Context Window: The maximum context length (e.g., 128K tokens) sets a hard upper bound for the total size of retrieved chunks plus the query and prompt.
• Embedding Model Capability: Most embedding models are optimized for chunks of a certain length (often 512-1024 tokens). Performance degrades for texts far outside this range.
• Retrieval Latency & Cost: Indexing and searching a million fine-grained chunks is more computationally expensive than searching 10,000 coarse chunks.
• Query Type: Simple keyword lookups benefit from fine chunks; complex analytical questions need coarser chunks.

Semantic chunking splits text based on natural content boundaries like paragraphs, topics, or entities, rather than arbitrary character counts. This strategy aims to create chunks that are semantically coherent, which can improve retrieval precision by ensuring each chunk represents a complete idea.

• Primary Use: Ideal for prose-heavy documents (reports, articles) where meaning is grouped in paragraphs.
• Contrast with Granularity: While granularity defines the size (sentence vs. section), semantic chunking defines the method for finding the boundary, often resulting in variable-sized chunks.

Fixed-length chunking segments text into chunks of a predetermined, uniform size, measured in characters or tokens. It is a simple, deterministic strategy that ensures predictable chunk sizes for indexing.

• Primary Use: Effective for code, logs, or dense text where semantic boundaries are less clear or for ensuring uniform processing costs.
• Granularity Control: The chosen fixed length (e.g., 512 tokens) directly defines the coarse vs. fine granularity. Smaller fixed lengths create fine-grained chunks; larger lengths create coarse-grained ones.

Hierarchical chunking creates a multi-level structure of chunks (e.g., document, section, paragraph) to enable retrieval at different levels of granularity. This architecture allows systems to first retrieve a coarse parent chunk and then drill down into finer child chunks.

• Key Structure: Often implemented using parent-child chunks, where a larger 'parent' chunk contains smaller 'child' chunks.
• Granularity Relationship: This strategy explicitly manages multiple granularities within a single document, allowing the retrieval system to select the appropriate level based on query specificity.

Recursive character text splitting is a widely used algorithm that splits text using a hierarchy of separators (e.g., \n\n, \n, ., ) until chunks are within a desired size range. It attempts to preserve semantic structure while adhering to size constraints.

• Mechanism: It first tries to split on the primary separator (e.g., double newlines). If the resulting chunks are too large, it recursively splits on the next separator in the list.
• Granularity Impact: The final chunk size range and separator hierarchy directly determine the effective granularity, often producing a mix of sentence and paragraph-level chunks.

Chunk overlap is a technique where consecutive text chunks share a portion of their content (e.g., 10% of the chunk size) to preserve contextual continuity. This mitigates the risk of losing crucial information that falls at a chunk boundary.

• Purpose: Prevents context fragmentation, especially for fixed-length or recursive splitting, ensuring concepts that span an artificial boundary are still captured.
• Trade-off: Increases index size and can introduce redundancy, but is often critical for maintaining retrieval recall, especially with finer granularities.

Sentence window retrieval is a hybrid retrieval strategy where a fine-grained core sentence is embedded and retrieved, and its surrounding context window (e.g., the sentences before and after) is then included for the language model. It decouples retrieval granularity from context provision.

• Process: 1. Embed and retrieve individual sentences (fine granularity for precision). 2. For each retrieved sentence, fetch its surrounding context from the original document.
• Granularity Advantage: Achieves high precision via sentence-level retrieval while providing the language model with the broader context needed for coherent generation, optimizing both recall and context utility.