Context Eviction Policy

The First-In-First-Out (FIFO) policy evicts context in the order it was added, regardless of how often it has been accessed. It treats the cache as a simple queue.

• Mechanism: New entries are appended to the tail; the entry at the head (oldest) is evicted when space is needed.
• Use Case: Effective for processing linear data streams where historical context has diminishing value, such as log parsing or real-time sensor data feeds.
• Drawback: Can evict frequently accessed, important context if it was loaded early.

The Least Frequently Used (LFU) policy evicts the context that has been accessed the fewest number of times. It prioritizes retention of popular, commonly referenced information.

• Mechanism: Maintains a counter for each cache entry that increments on access. The item with the smallest count is evicted.
• Use Case: Suitable for agents with long-running tasks where core instructions, rules, or reference data (accessed repeatedly) must be preserved.
• Challenge: Can lead to 'cache pollution' where a once-frequently used but now irrelevant item blocks new entries.

The Random Replacement (RR) policy selects a victim for eviction at random from the cache. It is a simple, low-overhead strategy with no tracking of access patterns.

• Mechanism: Uses a pseudo-random number generator to select an index to evict.
• Use Case: Provides a statistical baseline and can be effective when access patterns are unpredictable or when minimizing management overhead is critical.
• Performance: While often worse than LRU/FIFO in benchmarks, its simplicity can be advantageous in high-throughput, distributed systems.

A Time-To-Live (TTL) policy evicts context based on elapsed time since its creation or last access, not just when the cache is full. It's often combined with other policies.

• Mechanism: Each entry has a timestamp and a predefined lifespan (TTL). A background process or check on access removes expired entries.
• Use Case: Critical for ensuring data freshness and preventing stale context from influencing agent decisions, such as in dynamic financial or news analysis agents.
• Implementation: Common in distributed caches like Redis, and can be layered atop LRU for hybrid management.

Cost-Aware or Semantic Eviction is a sophisticated policy that uses a scoring function to evict the context deemed least valuable for the current task, considering factors beyond mere recency or frequency.

• Mechanism: Scores context based on semantic relevance to the active query, computational cost to re-generate, or information density. The lowest-scoring item is evicted.
• Use Case: Advanced agentic systems where context utility varies dramatically. For example, evicting a verbose anecdote before a concise, critical data point.
• Example: A system might use the embedding similarity between a cached chunk and the current conversation turn as the relevance score for eviction decisions.

The context window is the fixed-size, sequential block of tokens that a transformer-based language model can attend to in a single forward pass. It acts as the model's working memory, with sizes ranging from 4K tokens for smaller models to over 1M tokens for specialized long-context models. This hard limit necessitates eviction policies to manage content flow.

The Key-Value (KV) Cache stores computed attention states for previous tokens to speed up autoregressive generation. Cache eviction is the direct mechanism triggered by a context eviction policy, removing specific KV pairs from GPU memory when the cache exceeds its allocated size. Common policies include:

• LRU (Least Recently Used): Evicts the tokens attended to longest ago.
• FIFO (First-In-First-Out): Evicts the oldest tokens in sequence order.
• Attention-based: Evicts tokens with the lowest aggregate attention scores.

Context compression is a proactive alternative or complement to eviction. Instead of discarding tokens, it reduces their footprint while aiming to preserve semantic meaning. Key techniques include:

• Summarization: Using an LLM to condense long passages.
• Distillation: Extracting only the most salient facts or entities.
• Selective Filtering: Removing tokens deemed irrelevant by a scoring model. This reduces the need for eviction by making more efficient use of the available token budget.

Sliding window attention is an efficient attention mechanism that imposes a structural eviction policy. The model only attends to a fixed window of the most recent tokens (e.g., the last 4096). As new tokens are added, older tokens automatically fall outside the window and are architecturally evicted from direct attention, providing constant memory cost for infinite-length sequences. Frameworks like StreamingLLM build upon this concept.

A Context Management API is a software abstraction that implements eviction policies and other window management strategies for application developers. Libraries like LangChain's ConversationTokenBufferMemory or LlamaIndex's memory modules handle the mechanics of tracking token counts, applying FIFO/LRU eviction, and integrating with summarization tools, freeing engineers from manual context wrangling.

Multi-turn context is the accumulated history of a conversation that must be managed within the context window. An eviction policy directly determines which parts of this dialogue history are retained. Poor policy choice can lead to catastrophic forgetting of early task instructions or user preferences. Effective systems often use a hybrid approach: evicting turn-by-turn dialogue with LRU while preserving a compressed system prompt or conversation summary.