Context Compression

This technique uses a secondary, often smaller, language model to generate a concise abstract of the original context. The goal is to distill key facts, decisions, and narrative flow into a fraction of the original tokens.

• Extractive Summarization: Selects and concatenates key sentences or phrases directly from the source text.
• Abstractive Summarization: Generates new sentences that paraphrase and condense the original meaning, offering higher compression but risking hallucination.
• Use Case: Compressing a long multi-turn conversation history before feeding it into the next agentic reasoning step.

Instead of compressing everything, this approach uses a relevance scoring mechanism to filter out tokens deemed non-essential for the immediate task. It operates on the principle that not all context is equally valuable.

• Attention-Based Scoring: Analyzes attention patterns from a previous pass to identify tokens with low contribution to the final output.
• Query-Based Filtering: Uses the current user query or agent objective to retrieve only the most semantically relevant chunks from a larger memory store, ignoring irrelevant history.
• Key Advantage: Can preserve critical details verbatim, avoiding summarization errors, but risks discarding subtly important information.

These are granular, token-level operations that remove or combine elements of the input sequence based on learned heuristics or gradients.

• Token Pruning: Eliminates tokens identified as having low attention scores or gradient norms during a preliminary evaluation pass.
• Token Merging: Groups adjacent tokens with similar embeddings or semantic roles and replaces them with a single representative token (e.g., [MERGED]).
• Technical Note: Often implemented within the model's inference pipeline itself (e.g., LLaMA Pro's pruning) rather than as a pre-processing step.

A fundamental dichotomy in compression strategy, balancing fidelity against compression ratio.

• Lossy Compression: Techniques like summarization and filtering discard some information. The goal is to retain semantic utility for the task, not perfect reconstruction. This is the most common approach for LLM context.
• Lossless Compression: Aims for perfect reconstruction of the original text from the compressed form. In LLM contexts, this is rarely used for the primary context window but is relevant for storing memory externally (e.g., using gzip on text before vectorization).
• Engineering Trade-off: Higher compression ratios (more tokens removed) almost always require accepting some degree of information loss, which must be managed to prevent task degradation.

Designed for real-time, unbounded data streams like continuous sensor feeds or lengthy dialogues. Compression happens incrementally to maintain a constant memory footprint.

• Sliding Window with Summarization: Maintains a fixed token window of recent data, periodically summarizing the outgoing segment and storing the summary in a separate long-term memory track.
• Architecture: Often coupled with frameworks like StreamingLLM to manage attention mechanics. The system must decide what to summarize, when to summarize, and how to integrate historical summaries.
• Challenge: Avoiding catastrophic forgetting where critical information from earlier in the stream is permanently lost due to over-aggressive compression.

Represents context using dense, lower-dimensional codes rather than discrete tokens. The original text is encoded into a fixed-size semantic vector that captures its meaning.

• Process: A bi-encoder or cross-encoder model generates an embedding for a text chunk. This embedding becomes the compressed representation.
• Usage: The embedding is stored or passed forward. To "decompress," the model uses the embedding to condition its generation or to retrieve the closest original text from a backup store if needed.
• Limitation: This is inherently lossy and irreversible without a lookup table. It excels for retrieval (finding relevant context) but less so for tasks requiring verbatim recall.

The context window is the fixed-size, sequential block of tokens a transformer model can attend to in a single forward pass. It is the fundamental architectural constraint that necessitates compression.

• Fixed Capacity: Acts as the model's "working memory."
• Measured in Tokens: Includes input prompt, system instructions, and generated output.
• Hard Limit: Exceeding it requires truncation, summarization, or other compression techniques.

Context summarization is a specific compression technique where a language model generates a concise abstract of longer content.

• Model-Based Compression: Uses a secondary LLM call to distill information.
• Preserves Semantics: Aims to retain key facts, decisions, and intent.
• Trade-offs: Introduces latency from the extra inference step and risks hallucination or information loss during the summarization process.

Context truncation is the brute-force method of discarding tokens (typically from the middle or beginning of a sequence) to fit within a token limit.

• Simple Heuristic: Often uses FIFO (First-In, First-Out) or a "middle-out" strategy.
• High Risk of Information Loss: Critical task instructions or early conversational turns can be permanently deleted.
• Baseline Technique: Serves as a performance baseline against which more intelligent compression is measured.

Semantic chunking is a preprocessing step for compression, where text is split into meaningful segments based on content boundaries rather than arbitrary token counts.

• Enables Selective Retrieval: Creates coherent units that can be individually scored for relevance.
• Improves Compression Fidelity: Allows compression algorithms to operate on logical units, preserving narrative flow or argument structure.
• Foundation for RAG: Essential for creating high-quality retrievable context for Retrieval-Augmented Generation.

The KV Cache is a performance optimization that stores computed key and value tensors for previous tokens during autoregressive generation.

• Reduces Compute: Eliminates redundant calculations for tokens already processed.
• Memory Footprint: The cache itself consumes memory, often requiring its own eviction policies when handling long sequences.
• Compression Interaction: Advanced compression techniques may involve selective caching or pruning of the KV Cache to manage its growth.

Context window optimization is the overarching engineering discipline of strategically managing the limited token budget. Compression is a primary tool within this practice.

• Holistic Strategy: Involves selecting, ordering, and compressing information for maximum utility.
• Goal-Oriented: Decisions are based on the specific task (e.g., coding, long-document QA, multi-turn chat).
• Technique Portfolio: Engineers apply a combination of compression, caching, retrieval, and intelligent eviction to solve the context limit problem.