Granularity Control

Granularity control relies on semantic boundary detection to identify natural topic shifts within a document. Instead of splitting text at arbitrary character counts, the system analyzes embedding similarity between adjacent sentences or paragraphs.

• A significant drop in cosine similarity signals a topic boundary
• This prevents mid-thought fragmentation and preserves chunk coherence
• Algorithms often use a similarity threshold parameter to tune sensitivity

Effective granularity control applies a hierarchical splitting logic that respects document structure. The system attempts to split at major structural markers first, then falls back to finer separators only when necessary.

• Priority order: Document → Section → Paragraph → Sentence → Token
• This is the foundation of recursive chunking strategies
• Ensures that chunks remain self-contained and logically complete

Granularity must be calibrated against the target model's context window—the maximum token span an LLM can process in a single forward pass. Chunks that are too large exceed the window; chunks that are too small lose essential context.

• Typical chunk sizes range from 256 to 2048 tokens
• The optimal size depends on the embedding model's training parameters
• Chunk overlap provides a buffer to preserve cross-boundary meaning

Granularity control directly governs the specificity vs. context trade-off in retrieval-augmented generation. Fine-grained chunks yield highly precise search results but risk missing broader context; coarse chunks provide rich context but introduce noise.

• Atomic chunks maximize precision for factoid queries
• Parent document retrieval solves the trade-off by indexing small chunks but returning larger context blocks
• The optimal granularity is query-dependent and domain-specific

Granularity control is not one-size-fits-all. Different content types demand distinct splitting heuristics to maintain semantic integrity.

• Code: AST chunking splits at function and class boundaries
• Legal documents: Structural chunking respects clause and section numbering
• Narrative text: Semantic chunking follows topic flow
• Tables: Row-level or logical group splitting preserves tabular relationships

Granularity control must preserve metadata enrichment across chunk boundaries. Each segment inherits attributes from its source document—such as title, author, date, and section heading—to enable filtered retrieval.

• Metadata enables scoped queries (e.g., 'search only within 2024 documents')
• Chunk attribution links generated responses back to source segments
• Lost metadata degrades provenance and citation accuracy