Prompt Compression

A parameter-efficient method that prepends a small set of learnable continuous vectors—virtual tokens—to the input embedding sequence. Instead of engineering discrete text, the model optimizes these vectors directly through gradient descent. The frozen LLM interprets these optimized embeddings as an abstract, compressed instruction. This technique can reduce a hand-crafted prompt of hundreds of tokens into fewer than 10 virtual tokens, achieving comparable performance on classification and generation tasks while drastically cutting inference costs.

A budget-aware, coarse-to-fine compression framework that uses a small language model (e.g., LLaMA-7B) as a compressor. It works in three stages:

• Budget Allocation: Dynamically assigns a per-word compression ratio based on entropy.
• Iterative Token-Level Compression: Segments the prompt and removes low-information tokens.
• Distribution Alignment: Fine-tunes the compressor to bridge the gap between the small compressor and the target LLM. LLMLingua can achieve up to 20x compression on long prompts with minimal performance loss.

A technique that removes redundant or non-salient sentences from a prompt by scoring each lexical unit's informativeness using a small language model. The process computes a self-information metric for each token and filters out sentences below a dynamic threshold. This preserves the core semantic structure while eliminating filler. In practice, Selective Context can reduce prompt length by over 50% on tasks like summarization and dialogue without retraining the target model.

Uses a strong teacher model to generate a compressed, high-fidelity summary of a complex instruction or few-shot prompt. The summary is then used as a drop-in replacement for the original prompt. This method excels at condensing verbose chain-of-thought examples and multi-step reasoning traces into concise, abstracted demonstrations. The distilled prompt retains the logical structure while removing verbatim redundancy, often achieving a 5-10x compression ratio.

Trains a model to compress an entire prompt into a small set of gist tokens—virtual tokens inserted after the prompt that encapsulate its meaning. During inference, the model conditions solely on these gist tokens, ignoring the original prompt text. This effectively caches the prompt's semantic content in a fixed, compact representation. Gist tokens can compress prompts of any length into a constant 1-2 token overhead, enabling near-zero additional cost for repeated system prompts.

An architectural approach where a model is trained to recursively compress long documents into short summary vectors. These vectors are generated by the model itself and can be prepended to subsequent segments, acting as a compressed memory. The model learns to pack relevant information into these vectors during pre-training on a corpus. At inference, the summary vectors serve as a lossy, compressed representation of the preceding context, enabling processing of sequences far exceeding the standard context window.