Content Chunking

Each chunk must function as a standalone, context-independent unit of meaning. A chunk should answer a single question or define a single concept without relying on surrounding text for disambiguation.

• Core Principle: If a chunk is retrieved in isolation, a user or LLM must understand it completely without reading the preceding or following chunks.
• Implementation: Avoid starting chunks with ambiguous pronouns like 'it,' 'this,' or 'they.' Always re-establish the primary entity by name at the start of the chunk.
• Example: Instead of a chunk starting with 'It requires significant compute,' the chunk should read 'Training a large language model requires significant compute.'

Chunk boundaries must align with natural topic shifts, not arbitrary character or word counts. This requires analyzing the semantic structure of the document to identify where a distinct subtopic begins and ends.

• Methodology: Use the document's inherent structure—headings, subheadings, and paragraph breaks—as primary boundary signals.
• Advanced Technique: Employ a sliding window with a semantic similarity threshold. When the cosine similarity between consecutive sentences drops below a defined threshold, a new chunk boundary is triggered.
• Benefit: This prevents the 'semantic bleed' that occurs when a chunk contains the tail end of one concept and the beginning of another, confusing the embedding model.

To mitigate the 'lost in the middle' problem where LLMs ignore context in the center of a long prompt, chunks should include a small, overlapping window of context from the previous chunk.

• Overlap Strategy: Append the last 1-2 sentences of the preceding chunk to the beginning of the current chunk. This provides a 'running start' for the retrieval model.
• Metadata Anchoring: Enrich each chunk's metadata with a link to its parent document, its position in the sequence (e.g., chunk_index: 4/12), and links to adjacent chunks. This allows the retrieval system to fetch surrounding context on demand.
• Key Distinction: Contextual overlap is a retrieval-time aid, not a replacement for semantic self-containment.

The document's original hierarchical structure must be encoded into each chunk's metadata to maintain navigational context. A chunk from a subsection should know it belongs to a specific chapter and document.

• Metadata Schema: Include fields like doc_title, section_h1, section_h2, and section_h3 in every chunk's payload.
• Retrieval Utility: This allows for hierarchical retrieval, where a system can first retrieve a high-level section and then drill down into specific chunks, or present the full structural breadcrumb to the LLM for better reasoning.
• Example: A chunk about 'Q-Learning' should carry metadata that it belongs to Document: RL Fundamentals > Section: Model-Free Methods > Subsection: Value-Based Learning.

Chunk size is a critical hyperparameter that balances semantic richness against retrieval precision. The optimal size is dictated by the embedding model's context window and the downstream LLM's attention mechanism.

• General Heuristic: A range of 256 to 512 tokens is often optimal for dense retrieval, but this varies by domain.
• Token Counting: Always define chunk size by token count, not word or character count, as this is the native unit for transformer models.
• Trade-off: Smaller chunks (128 tokens) yield higher precision but may lack context. Larger chunks (1024+ tokens) provide more context but can dilute semantic focus and reduce the relevance score for specific queries.

A chunk is more than its text; it is a data object. The associated metadata is what enables sophisticated, filtered retrieval strategies beyond simple vector similarity.

• Essential Fields: Every chunk should carry a universally unique identifier (UUID), a creation timestamp, source URL, and content type.
• Filterable Attributes: Add domain-specific tags like product_category, audience_type, technical_level, or last_reviewed_date. This enables pre-filtering in vector databases, drastically reducing the search space before a similarity search is performed.
• Security: Metadata can also include access control lists (ACLs) to ensure that sensitive chunks are only retrieved by authorized users or agents.