Context Eviction

The First-In, First-Out (FIFO) policy evicts the oldest tokens in the sequence first. It operates like a queue, where new tokens push out the earliest ones.

• Mechanism: As the context window fills, tokens from the beginning of the conversation or document are discarded to make room for new input.
• Use Case: Ideal for streaming applications where recent information is most relevant, such as real-time chat or live data feeds.
• Drawback: Can discard critical foundational instructions or system prompts if they are placed at the start, potentially causing the model to 'forget' its initial role.

The Least Recently Used (LRU) policy evicts tokens that have been least referenced by the model's attention mechanism in recent decoding steps. It is a more sophisticated, attention-aware strategy.

• Mechanism: Tracks the attention scores or access frequency for context segments. Tokens that have not been attended to in many generation steps are candidates for removal.
• Use Case: Effective for long documents where certain sections become irrelevant as the conversation focus shifts. It helps retain 'active' context.
• Implementation: Often used in managing the KV Cache, where evicting less-referenced key-value pairs frees GPU memory for new tokens.

This strategy uses a scoring function to evaluate and rank the information density or utility of each context segment, evicting the lowest-scoring tokens.

• Scoring Methods: Can be based on:
  • Attention Heatmaps: Identifying tokens with consistently low aggregate attention.
  • Semantic Salience: Using a smaller model to score the relevance of sentences or paragraphs to the current query.
  • Metadata Tags: Programmatically marking instructions, examples, or user queries for prioritized retention.
• Use Case: Critical for complex, multi-turn agentic workflows where system instructions, few-shot examples, and intermediate conclusions must be preserved over conversational fluff.

This hybrid strategy combines context eviction with context compression. A sliding window evicts old content, but not before summarizing it into a condensed representation retained in context.

• Mechanism: As the context window slides forward, the text about to be evicted is processed by a summarization function (either the main model or a dedicated one). The summary is prepended to the window, preserving high-level information.
• Use Case: Essential for long-context tasks like book analysis, multi-document legal review, or extended diagnostic sessions where maintaining a narrative thread is necessary.
• Benefit: Dramatically extends the effective context horizon beyond the hard token limit by maintaining a 'summary of past events'.

This programmatic approach uses special tokens or markup to define evictable blocks within the prompt, giving the developer precise control.

• Mechanism: Context is organized into sections wrapped with tags like <evictable>. A post-processing step or custom attention mask removes these entire sections based on policy (e.g., after N turns).
• Examples:
  • Evicting old Q&A pairs in a chat history after 10 turns.
  • Removing completed sub-task instructions in a chain-of-thought prompt.
  • Cycling through a bank of few-shot examples to keep the demonstration set fresh.
• Benefit: Provides deterministic, predictable control over context state, which is vital for reproducible, production-grade applications.

Advanced systems employ dynamic eviction where the policy itself changes based on real-time model behavior or task characteristics.

• Adaptive Triggers: Eviction is not just based on token count but on metrics like:
  • Perplexity Spike: A sudden increase in generation uncertainty may signal missing context, triggering a review of eviction decisions.
  • Tool Call Frequency: In an agentic loop, reduced tool usage might indicate forgotten instructions, prompting a partial context reset.
  • Query Type Detection: A shift from 'summarization' to 'detailed QA' may trigger a switch from a FIFO to an LRU policy.
• Use Case: Found in sophisticated Agentic Memory and Context Management systems, where context is treated as a dynamic resource to be optimized for task performance.

When analyzing legal contracts, research papers, or codebases exceeding the context limit, a sliding window with eviction is used. The system processes the document in segments, but must decide which prior segments to retain as working memory. For example, when reviewing a 100-page contract, the model might evict details of early boilerplate sections to keep the most relevant clauses and definitions active for cross-referencing in later analysis.

Chatbots and virtual assistants for customer support or technical troubleshooting must maintain context over long, multi-turn dialogues. A Least-Recently-Used (LRU) eviction policy can be applied to the conversational history stored in the KV cache. This ensures the most recent exchanges and the core user intent remain, while older, tangential details are removed to prevent the context from being filled with obsolete information, which can degrade response relevance.

In agentic workflows where an AI performs a sequence of actions (e.g., "research a topic, draft a report, create slides"), each step may require different context. After completing the research phase, the detailed source texts might be summarized and evicted to make room for the draft outline and slide content. This task-aware eviction prioritizes information needed for the current subtask, preventing context pollution from prior stages.

Systems processing live feeds—like financial tickers, sensor telemetry, or social media streams—use eviction to maintain a rolling context window. As new data arrives, the oldest data points are evicted. The eviction policy must be tuned to the data's volatility; for high-frequency trading, recent price action is critical, so eviction is aggressive. For trend analysis, a larger window of aggregated historical data might be prioritized and older raw ticks evicted.

When an AI pair-programmer iteratively writes and refines code, the context contains the code file, error messages, and user instructions. As the file grows, selective eviction of earlier, stable code sections (replaced with high-level comments) frees tokens for new code and runtime feedback. The system must evict intelligently to avoid removing function signatures or variable definitions still in scope, which would break the program's coherence.

In advanced RAG systems, after retrieved documents are used to answer a query, the full text of those documents may be evicted from the primary context to conserve space for follow-up questions or synthesis. Instead, a dense vector representation (embedding) of the answer's provenance is retained. This allows the system to recall and re-retrieve details if needed, without storing the entire source text in the limited context window.

The context window is the fixed-size, contiguous block of tokens a transformer model can process in a single forward pass, serving as its working memory. It is a fundamental architectural constraint determined by the model's training and the available KV cache memory during inference.

• Defines the absolute upper limit for input + output tokens.
• Exceeding it requires eviction, truncation, or chunking strategies.
• Sizes range from 4K tokens in older models to 1M+ tokens in modern, long-context variants.

The KV Cache is a critical performance optimization in transformer decoders. During autoregressive generation, it stores computed key and value vectors for all previous tokens, preventing redundant computation for each new token generation.

• Its memory footprint is proportional to context length and model size.
• Cache eviction policies are required to manage this memory during long sequences.
• Directly limits practical context length independent of the model's theoretical window.

Context compression is a proactive strategy to reduce token count before hitting hard limits, aiming to preserve semantic content. It is a higher-level alternative or complement to blunt eviction.

• Techniques include summarization, selective pruning, and redundancy elimination.
• Goal is to increase information density within the fixed window.
• Contrasts with eviction, which simply removes tokens based on position or score.

Context truncation is the simplest, most lossy form of context management: removing tokens from the beginning or end of a sequence to meet a token limit. It is a primitive form of eviction with a fixed policy (remove oldest or newest).

• Commonly applied automatically by APIs when input exceeds the limit.
• Often degrades performance on long-context tasks.
• Serves as a baseline against which more sophisticated eviction and compression strategies are measured.

Sliding window attention is an architectural sparse attention pattern that enforces a local context window. Each token can only attend to a fixed number of preceding tokens, enabling efficient processing of sequences far longer than the training context.

• Models like Mistral 7B use this to extrapolate to long sequences.
• Creates a natural, architecture-enforced form of context eviction for tokens outside the local window.
• Reduces computational complexity from quadratic to linear for long sequences.

Cache eviction specifically refers to the algorithms used to manage the GPU memory allocated for the KV Cache. When generating very long sequences, the cache can exceed GPU memory, requiring parts of it to be discarded.

• Policies include Least-Recently-Used (LRU) or evicting the middle segments of long contexts.
• A low-level system concern that directly enables or constrains high-level context eviction strategies.
• Essential for maintaining inference speed and stability during long dialogues or document generation.