Context Compression

The process of algorithmically condensing long passages of text into shorter, information-dense representations. This is a lossy compression technique that aims to preserve core semantic meaning and key facts.

• Methods: Can be performed by the LLM itself (e.g., "Summarize the previous conversation") or by a separate, smaller model.
• Use Case: Essential for long-running chat sessions or document analysis where historical context must be retained but in a condensed form.

The policy-driven removal of specific tokens or segments from the active context window. Unlike truncation, it can target less relevant middle sections.

• Policies: Can be based on recency (drop oldest tokens), relevance scoring (drop tokens with low attention weight), or task-specific heuristics.
• Example: In a long code review, evicting early, resolved comments to keep recent feedback and the latest code diff in context.

Identifies and removes duplicate or highly similar information within the context. This is a lossless compression technique for semantic content.

• Mechanism: Uses embedding similarity or exact string matching to find repeated phrases, list items, or factual statements.
• Impact: Directly increases information density by freeing token budget wasted on repetition. Crucial when aggregating content from multiple similar sources.

Converting information within a context window into a dense vector representation for later retrieval, rather than keeping the raw tokens active.

• Process: A summary or key facts are encoded into an embedding and stored in a vector database. When relevant, a similar query retrieves this compressed context.
• Benefit: Enforms a form of dynamic context, where a vast amount of "cold" information is stored externally and only a relevant, compressed subset is injected into the primary window.

Advanced methods that use trained models to replace token sequences with shorter, learned representations. This goes beyond simple summarization.

• Examples: Landmark attention (representing long segments with single landmark tokens), or perceiver-style architectures that compress a long context into a fixed-size latent bottleneck.
• Characteristic: These are often model-architecture changes rather than post-processing techniques, requiring specialized training.

Organizes information into a structured hierarchy (like a tree or outline) where only the high-level nodes are kept in immediate context, with details stored externally.

• Operation: The LLM interacts with a structured representation. Details are fetched on-demand via function calls when the model "drills down."
• Analogy: Similar to how an operating system uses RAM and disk; the context window holds the "working set," while a knowledge graph or database holds the full structured corpus.

Enables models to process books, legal contracts, or lengthy reports by compressing earlier sections into dense summaries. This preserves the narrative or argumentative thread while freeing tokens for new analysis.

• Key Technique: Recursive summarization, where the model summarizes a chunk, and that summary becomes part of the context for summarizing the next chunk.
• Example: Analyzing a 100-page technical manual by maintaining a running summary of each chapter, allowing the model to answer questions about the document's entirety.

Maintains coherent, long-term interactions with users by compressing the conversation history. Without compression, the context window fills with past exchanges, causing the model to 'forget' early context.

• Key Technique: Context summarization applied to dialogue turns. The system might periodically replace the full message history with a brief summary (e.g., 'User interested in pricing for Enterprise plan. Discussed basic features.').
• Benefit: Enables support bots, therapeutic chatbots, and creative co-pilots to have sessions lasting hours or days without hitting token limits or incurring prohibitive costs.

Optimizes the use of retrieved documents in RAG pipelines. Instead of concatenating multiple full-length source documents into the context, compression extracts only the most relevant snippets or synthesizes a single answer-bearing context.

• Key Techniques: Redundancy elimination across retrieved chunks and extractive or abstractive compression of individual documents.
• Impact: Reduces noise, decreases latency, and lowers cost by passing fewer tokens to the Large Language Model while improving answer precision by focusing on salient information.

Directly addresses the economic and performance constraints of using Large Language Models. Inference cost and time are often proportional to the number of input tokens processed.

• Mechanism: By applying selective pruning or summarization to user queries, system instructions, or few-shot examples, the total token count is reduced.
• Result: Enables higher throughput, lower per-request cost, and faster response times, which is critical for scaling AI applications to millions of users. A 30% reduction in context length can lead to a nearly proportional reduction in inference cost.

Allows models to analyze continuous feeds of data (e.g., news, social media, financial tickers, IoT sensor logs) by maintaining a compressed, rolling window of the most significant events.

• Key Technique: Dynamic context eviction and prioritization, where older, less critical data points are summarized or dropped as new data arrives.
• Application: Used in algorithmic trading for sentiment analysis, in security operations centers for threat detection, and in industrial settings for predictive maintenance, where only the recent, anomalous, or high-signal events need to be retained in active context.

Supports autonomous agents that perform complex, multi-step workflows involving planning, tool calls (API execution), and reflection. Each step (reasoning, tool result, error message) consumes context.

• Challenge: Without compression, an agent's own lengthy internal monologue and tool outputs exhaust its context window, halting complex tasks.
• Solution: Strategic summarization of past reasoning steps and tool results. The agent compresses 'what was done and learned' into a concise state, preserving the goal and key findings while reclaiming tokens for next steps. This is fundamental to ReAct and similar agentic frameworks.

The context window is the fixed-size, contiguous block of tokens a transformer model can process in a single forward pass. It acts as the model's working memory for a given input and output sequence. Its size is a fundamental architectural constraint that dictates how much information the model can consider at once.

• Fixed Capacity: Determined during model pre-training (e.g., 128K tokens).
• Input & Output Shared: The limit includes both the prompt and the generated response.
• Bottleneck: Long documents or multi-turn conversations require strategies like compression to stay within this limit.

The KV Cache is a critical inference optimization for autoregressive transformers. It stores computed key and value vectors for previous tokens during generation, preventing their re-computation for each new token.

• Speed vs. Memory Trade-off: Drastically improves generation latency but consumes GPU memory proportional to context length.
• Compression Target: In long contexts, the KV cache itself can become a memory bottleneck, leading to techniques like cache eviction or quantization.
• Fundamental Mechanism: Enables efficient attention over long sequences by caching intermediate states.

Context summarization is an active compression technique where a long passage within the context is condensed into a shorter, information-dense representation. This is often performed by the model itself via a dedicated instruction.

• Abstractive vs. Extractive: Can generate new summary text or select key sentences.
• Agentic Applications: Crucial for long-running conversational agents to maintain a distilled history.
• Lossy Process: Inevitably loses some granular detail, requiring strategic application.

Redundancy elimination is a compression technique that identifies and removes duplicate or highly similar information within the context window. It operates on the principle that token budget should be reserved for unique semantic content.

• Lexical vs. Semantic: Can target exact token repeats or paraphrased concepts with similar meaning.
• Algorithmic Approaches: Uses embeddings and similarity scores to find near-duplicates.
• Information Density: Directly increases the information density (utility per token) of the remaining context.

Sparse attention is an architectural approach to context management, not a runtime compression technique. It designs the transformer's self-attention mechanism to only compute a subset of all possible token-to-token connections.

• Complexity Reduction: Aims to change the fundamental quadratic cost O(n²) of attention to near-linear O(n log n).
• Patterns: Includes sliding window attention (local context) and dilated or global patterns.
• Enables Longer Contexts: Models like Longformer or GPT-4 variants use sparse attention to support context windows in the hundreds of thousands of tokens.

Context chunking is a processing strategy, not compression, that divides a document longer than the context window into smaller, sequential segments. Each chunk is processed independently or with some overlapping state.

• Map-Reduce Pattern: Often used for summarization or Q&A over long texts, where results from each chunk are combined.
• Loss of Cross-Chunk Attention: The primary limitation is that the model cannot draw direct connections between tokens in non-overlapping chunks.
• Pre-processing Step: A prerequisite for applying other techniques (like summarization) to each manageable segment.