Contextual Compression

Unlike generic summarization, contextual compression is strictly query-dependent. A lightweight compressor model evaluates each retrieved document chunk against the user's specific question, extracting only the spans, sentences, or facts that are semantically relevant. This process discards the surrounding noise, ensuring that every token placed in the context window directly contributes to answering the query. The compressor acts as a filtering gate, mapping (query, document) -> compressed_document.

The primary goal is to overcome the Lost-in-the-Middle problem and token limits. By compressing 10 retrieved documents into information-dense snippets, the system can effectively present the model with the key facts from a much larger corpus. This transforms the context window from a passive container into an active, high-signal workspace, allowing the LLM to reason over a broader set of sources without exceeding its architectural constraints.

The compressor is typically a smaller, efficient model fine-tuned for extractive or abstractive compression:

• Extractive Compression: Selects and concatenates the most relevant sentences from the source document.
• Abstractive Compression: Generates a condensed summary in new, concise language.
• Encoder-Based: Models like BERT are often used for token-level relevance classification.
• Seq2Seq: Models like T5 can be trained to output a compressed string given the query and document as input.

Contextual compression sits as a post-retrieval, pre-generation stage in a RAG pipeline. The standard flow is:

1. Retrieval: Fetch top-k document chunks from a vector store.
2. Compression: Pass each chunk through the compressor with the query.
3. Assembly: Concatenate the compressed results into the final prompt. This ensures the LLM sees a high-density information payload rather than raw, noisy chunks.

By stripping away irrelevant context, compression reduces the cognitive load on the LLM. The model is less likely to be distracted by tangential information or to conflate facts from different sources. The compressed content provides clean, atomic facts that are directly tied to the query, which improves factual grounding and lowers the probability of the model generating plausible-sounding but incorrect statements based on misinterpreting noisy context.

Introducing a compressor adds a computational step, increasing latency. However, this is often offset by the reduction in prompt tokens sent to the larger, more expensive LLM. The trade-off is a cost-performance optimization:

• Small Compressor: Fast, low-cost filtering.
• Large Generator: Receives fewer, higher-quality tokens. The net effect is often a reduction in total inference cost and time-to-first-token, especially when the initial retrieved documents are large.

A set of techniques that condense a lengthy prompt into a smaller, information-dense representation to reduce token usage, latency, and cost without sacrificing essential semantic content.

• Extractive compression: Removes redundant or non-essential tokens from the original text
• Abstractive compression: A smaller model rewrites the prompt to convey the same meaning in fewer tokens
• Soft prompting: Replaces discrete text instructions with learned continuous vector embeddings
• Directly reduces the load on the KV-Cache and accelerates time-to-first-token

A documented performance failure mode of LLMs where information placed in the center of a long context window is significantly less likely to be retrieved or utilized compared to information at the beginning or end.

• Models exhibit a strong U-shaped attention curve over position
• Information at the start benefits from the primacy effect; information at the end benefits from the recency effect
• Critical for RAG architects: retrieved documents should be placed at the extremes of the prompt
• Contextual compression mitigates this by reducing total document length, pushing key facts toward the edges

A second-pass retrieval stage where a more computationally intensive, high-precision model re-scores an initial set of candidate documents to ensure the most semantically relevant chunks are placed at the top of the context window.

• Two-stage pipeline: Fast vector search (bi-encoder) retrieves candidates; slow cross-encoder re-ranks them
• Maximizes the probability that the most critical information occupies the high-attention primacy region
• Often paired with contextual compression to form a retrieve-then-compress architecture
• Popular implementations include Cohere Rerank, BGE-Reranker, and RankLLM

A content segmentation strategy that divides a document into chunks based on semantic boundaries—such as paragraph breaks or topic shifts determined by embedding similarity—rather than a fixed character or token count.

• Uses cosine similarity between consecutive sentences to detect natural topic transitions
• Produces self-contained, coherent chunks that are ideal for precise retrieval
• Prevents the fragmentation of a single concept across multiple chunks
• When combined with contextual compression, ensures the compressor receives well-formed, logically complete input units

A memory buffer that stores the pre-computed key and value tensors from previous decoding steps in a Transformer, eliminating redundant computation and accelerating autoregressive text generation.

• During the prefill phase, keys and values are computed for the entire input prompt and cached
• During the decode phase, only the new token's query, key, and value are computed; past keys/values are read from cache
• The cache size grows linearly with sequence length, making it a primary memory bottleneck for long contexts
• Contextual compression reduces the prompt length, directly shrinking the KV-Cache footprint and enabling higher throughput

An attention optimization that uses a single shared key-value head for multiple query heads, significantly reducing the size of the KV-Cache and memory bandwidth requirements while maintaining quality close to multi-head attention.

• Multi-Head Attention (MHA): Each query head has its own K and V head — high memory cost
• Multi-Query Attention (MQA): All query heads share a single K and V head — memory efficient but quality loss
• GQA is the intermediate sweet spot: query heads are divided into groups, each group sharing one K-V head
• Adopted by Llama 2, Llama 3, and Mistral; works synergistically with contextual compression to maximize usable context length