Prompt Compression

The core mechanism of tools like LLMLingua involves calculating the perplexity of each token in the original prompt using a small, fast language model. Tokens with low perplexity—those the model predicts with high confidence—are deemed non-essential or redundant. The compression algorithm then removes these high-confidence, low-information tokens, retaining only the high-perplexity tokens that carry the most semantic weight and uncertainty. This process directly reduces the total token count without requiring the large target model to process the full original text.

Advanced prompt compressors allow for a dynamic token budget, where the compression rate is conditioned on the specific instruction or context segment. The system can be instructed to apply aggressive compression to a lengthy background document while preserving the integrity of a complex, multi-step instruction verbatim. This is achieved through segment-level granularity, where different compression ratios are applied to distinct parts of the prompt based on their functional role. This ensures that the model's task definition remains pristine while reference material is heavily condensed.

Rather than a single-pass removal, some compression methods use an iterative distillation loop. A small model proposes a compressed prompt, which is then fed to the target model. The output is compared against the output from the full, uncompressed prompt. A contrastive loss function then provides a signal back to the compressor, teaching it which tokens were critical for maintaining the exact output distribution. Over multiple iterations, this aligns the compressed prompt's effect with the original, ensuring high task fidelity even at extreme compression rates.

Beyond token removal, context distillation compresses a long prompt into a small set of virtual tokens or a learned embedding vector. Instead of producing a human-readable shortened prompt, the system trains a small set of soft prompts that, when prepended to the actual query, elicit the same behavior from the frozen target model as the full original instruction. This method achieves extreme compression by operating in the model's continuous latent space, bypassing the need for discrete text altogether.

A critical failure mode of naive compression is the removal of named entities that are central to the task. Sophisticated compressors integrate a salience estimation module that first identifies key entities, dates, and numerical values. A preservation mask is then applied during the token removal process to ensure these high-salience entities are never pruned, even if their individual perplexity is low. This guarantees that the compressed prompt retains the specific nouns and facts required for accurate, grounded generation.

This approach combines the safety of extractive methods with the flexibility of abstractive ones. The system first performs extractive compression by selecting the most salient sentences. It then applies a lightweight abstractive model to paraphrase and fuse those sentences, further reducing length while smoothing over the disjointed text that pure extraction can cause. This hybrid pipeline ensures the output is both factually grounded in the source text and linguistically fluent, avoiding the choppy, fragmented output of extractive-only methods.

The process of compressing a large, complex prompt or set of instructions into a smaller, more efficient set of soft prompts or vectors that elicit identical behavior from a language model. Unlike hard prompt compression which removes tokens, distillation trains a compact representation that the model interprets as equivalent to the original lengthy instruction, dramatically reducing per-request token consumption.

A documented phenomenon where large language models exhibit a significant performance drop when retrieving and using information located in the middle of a long context window, favoring the beginning and end. This positional bias directly motivates prompt compression—by reducing overall length, critical information can be repositioned into high-attention zones, ensuring it is not ignored during generation.

A prompting technique that iteratively refines a summary to increase its information density, packing more entities and details into a fixed-length output without sacrificing clarity. CoD complements prompt compression by demonstrating how to maximize semantic payload per token. The iterative process:

• Start with a sparse summary
• Identify missing salient entities
• Fuse entities into increasingly dense sentences
• Stop when readability degrades

A content segmentation strategy that splits documents into chunks based on semantic similarity rather than a fixed character count. By ensuring each chunk contains a coherent, self-contained topic, semantic chunking reduces the need for redundant bridging context in prompts. This pre-processing step works synergistically with compression—well-chunked content compresses more cleanly because each segment already represents a discrete, complete idea.

An algorithm for information retrieval and summarization that selects passages by balancing their relevance to a query against their similarity to already-selected content. MMR serves as a pre-compression filter, eliminating redundant information before it ever enters the prompt. This reduces the token load that compression techniques must later process, creating a two-stage efficiency pipeline.