Information Density

This is the core measure of information density. High-density text conveys its meaning with fewer, more precise tokens. For example, the sentence 'The algorithm's quadratic time complexity necessitates optimization' is more information-dense than 'The way the program works takes a lot more time as the input gets bigger, so we need to make it faster.' The former uses domain-specific terminology (quadratic time complexity) to pack meaning into fewer tokens, preserving valuable context window space for other instructions or data.

A model's context window is a finite resource. High information density maximizes the amount of useful instruction, knowledge, and examples that can fit within this limit.

• Low Density: Fills the window with verbose, repetitive, or filler text, leaving less room for critical task specifications or relevant data.
• High Density: Allows for more comprehensive system prompts, more few-shot examples, or longer document excerpts within the same token budget, leading to better-informed and more reliable model outputs.

Many context compression strategies aim to increase information density. Techniques like summarization, redundancy elimination, and extractive pruning work by distilling the original text into a shorter form that retains the core semantic content. The success of these techniques is measured by how well they preserve the original information's utility (high retained density) while reducing its token footprint. Poor compression loses critical nuance, effectively lowering the functional density of the compressed text.

Maximizing density often involves a trade-off. Excessively dense text—filled with jargon, acronyms, or overly terse phrasing—can become ambiguous or difficult for the model to parse correctly, leading to misinterpretation. The goal is optimal density: using precise, unambiguous language without unnecessary elaboration. For instance, in a system prompt, 'Format output as JSON' is optimally dense; 'You should make the thing you give back be in this format called JSON' is low-density; and 'Output JSON' may be too dense, lacking critical instruction about what data to include.

The ultimate test of information density is downstream model performance. In prompt engineering, two prompts achieving the same task outcome are compared: the one that does so with fewer tokens (higher density) is superior, as it frees up context for other uses. In Retrieval-Augmented Generation (RAG), the density of retrieved passages affects answer quality; a dense, relevant chunk provides better grounding than a verbose one that dilutes key facts. Density is thus an operational metric, evaluated by its effect on accuracy, reliability, and cost-per-inference.

Information density is a qualitative property distinct from simple token count. A 100-token passage of highly repetitive marketing copy has very low information density. Conversely, a 100-token passage from a technical specification or a well-crafted few-shot example may have very high density. Context truncation reduces token count but does not increase density—it simply discards information. True optimization focuses on increasing the value (density) of the tokens that remain within the token limit, making every token count.