Prompt Compression

The core mechanism of prompt compression is maximizing information per token. This involves removing linguistic redundancy, filler words, and non-essential context that do not contribute to the model's ability to perform the target task. Effective compression identifies and preserves only the salient semantic units and instructional logic required for accurate generation, often achieving 2x-5x reduction in prompt length without measurable performance degradation.

A parameter-efficient technique where a small set of learnable virtual tokens is prepended to the input. These continuous vectors are optimized via gradient descent to capture task-specific context, effectively compressing lengthy instructions into a compact, tunable prefix. Unlike hard prompt engineering, soft prompts exist in the model's embedding space and can condense thousands of tokens of few-shot examples into a handful of learned parameters, dramatically reducing per-query costs.

A selective compression strategy that removes entire sentences, paragraphs, or documents from the prompt based on their information salience ranking. Techniques include:

• LLMLingua: Uses a small language model to score and remove low-importance tokens
• Selective Context: Dynamically drops content below a self-assessed relevance threshold
• Recursive summarization: Iteratively condenses long documents before insertion This approach is critical for retrieval-augmented generation pipelines where retrieved chunks often contain redundant or marginally relevant information.

Rather than truncating or pruning, abstractive compression uses a smaller, faster model to rewrite the prompt into a condensed form. The compressor model generates a dense 'gist' that preserves the original intent, key entities, and required reasoning steps. This is distinct from summarization—the goal is not a human-readable summary but a machine-optimized representation that maintains the original prompt's functional utility for the downstream generator model.

Leverages the Key-Value (KV) cache of transformer models to avoid re-encoding repetitive prompt prefixes. Systems like PromptCache and AttentionStore identify overlapping content across queries and reuse cached attention states. For applications with shared system prompts or frequently accessed documents, this can reduce time-to-first-token by 50-80% and eliminate redundant compute, effectively compressing the incremental cost of each new query to only its novel components.

A compression method that removes tokens with low information entropy—those that are highly predictable from surrounding context and contribute minimally to the model's uncertainty reduction. By analyzing the self-information of each token in the prompt, compressors can surgically excise predictable function words, common phrases, and boilerplate while preserving high-entropy content like named entities, numbers, and domain-specific terminology that are critical for factual accuracy.

A budget-aware compression framework that uses a small language model (e.g., GPT-2 Small) to dynamically remove non-essential tokens from prompts. It calculates perplexity-based token importance to identify and retain only the most information-dense elements.

• Token-level pruning removes low-information words and phrases
• Maintains task performance at compression ratios up to 20x
• Integrates with black-box LLMs without requiring access to internal weights
• Particularly effective for in-context learning with many examples

A method that compresses prompts by iteratively evaluating each sentence's conditional information entropy relative to the query. Sentences with low mutual information are discarded before model invocation.

• Uses self-information scoring to rank sentence utility
• Reduces context window consumption without summarization
• Preserves exact source wording for citation grounding
• Compatible with both dense and sparse retrieval pipelines

An architectural approach where a model is trained to produce summary vectors—compressed soft representations of long documents—that can be prepended to prompts as virtual tokens. These vectors are optimized end-to-end for downstream task performance.

• Generates fixed-length soft prompts from variable-length inputs
• Summary vectors are passed across model calls for multi-hop reasoning
• Eliminates the need for explicit extractive summarization
• Requires fine-tuning but yields superior information density

A technique that trains a model to compress lengthy system instructions into a small set of gist tokens that condition the model's behavior identically to the full prompt. Once trained, these gist tokens replace verbose instructions entirely.

• Compresses instructions by up to 26x
• Gist tokens are task-specific and reusable across queries
• Reduces per-query token costs in production deployments
• Demonstrated on models like FLAN-T5 with minimal accuracy loss

A compression strategy that identifies and removes attention heads in transformer models that contribute minimally to retrieval-augmented tasks. By pruning heads focused on syntactic rather than semantic features, the effective context processing is streamlined.

• Uses head importance scoring via Taylor expansion
• Reduces computational cost without retraining
• Particularly effective for long-context retrieval scenarios
• Complements token-level compression for compound savings

A method that replaces numerous in-context learning demonstrations with a single compact task vector derived from the difference in model activations when processing examples. This vector encodes the task pattern without storing full exemplars.

• Compresses dozens of examples into one vector representation
• Task vectors can be pre-computed and cached
• Enables few-shot performance at zero-shot latency
• Applicable to classification, translation, and reasoning tasks

A set of techniques designed to counteract the tendency of language models to ignore or under-attend to information placed in the center of a long context window. Prompt compression often serves as a mitigation strategy by distilling critical information regardless of its original position.

• Core Problem: Models exhibit a U-shaped attention curve, focusing on the beginning and end of prompts.
• Compression Link: By extracting salient entities and relationships, compression ensures mid-context data is repositioned or condensed into high-attention zones.
• Techniques: Involves strategic re-ordering of documents and hierarchical summarization before final generation.

The phenomenon where a language model in a multi-turn conversation gradually loses focus on the original user intent or key facts as the dialogue history grows longer. Prompt compression acts as a state reset mechanism.

• Mechanism: Irrelevant generated tokens dilute the signal of the original instructions.
• Compression Role: Periodically compressing the conversation history into a dense running summary prevents the model from fixating on tangential topics.
• Result: Maintains high factual consistency and instruction adherence over extended sessions.

The computational task of assigning importance scores to pieces of information to identify the most relevant content for inclusion in a summary. This is the prerequisite step for effective prompt compression.

• Query-Focused: Salience is often calculated relative to the user's query, not just the document's central theme.
• Compression Pipeline: A salience ranker acts as a filter before the compressor, discarding low-score tokens to preserve only high-density information.
• Metrics: Uses attention weights, gradient-based feature attribution, or dedicated cross-encoder models.

A greedy algorithm for information retrieval and summarization that balances the relevance of a sentence to a query with its novelty relative to already-selected sentences. It is a classic, non-parametric compression logic.

• Redundancy Elimination: Explicitly penalizes sentences that are too similar to the current summary set.
• Lambda Parameter: A tunable hyperparameter controlling the relevance-novelty trade-off.
• Modern Use: Often replaced by neural models, but the core principle of diversity maximization remains central to preventing compressed prompts from being repetitive.

An automated metric that quantifies the degree to which a generated summary's factual assertions align with the information contained in the source documents. This is the primary evaluation guardrail for lossy compression.

• Entailment Check: Uses Natural Language Inference (NLI) to verify if the compressed prompt logically follows from the original.
• Compression Risk: High compression ratios increase the risk of introducing hallucinations or dropping critical negations.
• Implementation: Typically involves a separate NLI model scoring the compressed output against the original text chunks.

The broader discipline of reducing compute costs during model execution. Prompt compression is a pre-inference optimization that directly reduces time-to-first-token (TTFT) by shrinking the input payload.

• Cost Correlation: Transformer attention complexity scales quadratically with input length; compression reduces this polynomial cost.
• Complementary Techniques: Works synergistically with KV-cache optimization and speculative decoding.
• Business Impact: Directly translates to lower GPU utilization and faster user-facing response times.