Chunk Information Density

The primary metric for chunk information density, calculated as the number of discrete, verifiable facts divided by the total token count. A high ratio indicates efficient encoding.

• Target: Maximize unique facts per token
• Anti-pattern: Filler phrases, redundant summaries, or verbose transitions
• Example: A chunk stating 'The Eiffel Tower is 330 meters tall and was completed in 1889' has a higher ratio than one padded with 'It is interesting to note that the famous landmark known as the Eiffel Tower...'

High-density chunks exhibit low semantic entropy, meaning each token contributes directly to the core meaning. Removing any token should measurably degrade the chunk's informational completeness.

• Principle: Every word must earn its place
• Technique: Prune hedging language ('may possibly', 'it could be argued')
• Validation: Use embedding similarity to verify that a pruned chunk remains semantically equivalent to the original

Dense chunks concentrate on a small number of highly salient named entities rather than diffusing attention across many peripheral concepts. This improves retrieval precision for entity-specific queries.

• Optimal count: 1-3 primary entities per chunk
• Method: Extract entities via NER; if count exceeds threshold, split the chunk
• Benefit: Higher cosine similarity scores for targeted vector searches

A quantitative measure of how much new knowledge a chunk provides beyond what a language model already knows from pre-training. High-density chunks score highly on novelty.

• Calculation: Compare chunk content against model's parametric knowledge baseline
• High-gain signals: Proprietary data, recent statistics, unique case studies
• Low-gain signals: Common knowledge, widely documented facts, generic definitions

Profiling the distribution of token types within a chunk to identify and eliminate low-value tokens that consume context window space without adding information.

• High-value tokens: Proper nouns, technical terms, numerical values, dates
• Low-value tokens: Discourse markers ('furthermore', 'however'), redundant modifiers
• Tooling: Use token classification models to score each token's contribution

Measuring how much a chunk can be losslessly compressed while retaining its factual payload. Higher compressibility indicates lower initial density and the presence of linguistic redundancy.

• Method: Apply text summarization; compare fact count before and after
• Ideal: Near-zero compression possible without fact loss
• Benchmark: If a 200-token chunk can be reduced to 120 tokens with all facts intact, the original density was suboptimal