Context Caching

The KV Cache is the foundational state cached during autoregressive generation in transformer models. For each layer and attention head, the model computes Key (K) and Value (V) tensors for every input token. During sequential token generation, these tensors for previously generated tokens are stored and reused, avoiding the quadratic recomputation cost of full attention over the growing sequence. This provides a near-constant time per-token generation after the initial prompt, making it the primary driver of inference latency reduction. Its size grows linearly with batch size, sequence length, and model dimensions.

Because the KV Cache consumes GPU memory, eviction policies are required to manage its growth within hardware limits. These policies determine which cached tokens are removed first when the cache is full.

• Least Recently Used (LRU): Discards the tokens that have been attended to the least in recent generation steps. This is common for conversational agents where recent dialogue is most relevant.
• First-In-First-Out (FIFO): Evicts the oldest tokens (e.g., the initial prompt) first. This is simpler but can discard critical foundational context.
• Attention-Score-Based: Removes tokens with the lowest aggregate attention scores, theoretically preserving the most "important" context. Advanced frameworks like StreamingLLM identify and preserve attention sinks (initial tokens) to maintain generation stability during eviction.

Beyond the low-level KV Cache, a Conversation History Cache stores high-level dialogue turns (user queries and agent responses) in a structured, compressed format. This is typically managed by a Context Management API (e.g., LangChain's ConversationBufferMemory). Features include:

• Summarization: Periodically using an LLM to condense old dialogue into a concise summary, which is then cached as the conversation's "backstory."
• Semantic Indexing: Storing history chunks in a vector database for semantic retrieval, allowing the agent to pull in relevant past exchanges based on the current query's meaning, not just recency.
• This cache operates at the application level, providing semantic continuity without always consuming precious context window tokens.

A Sliding Window Cache is an implementation of the sliding window attention mechanism, where the model's attention and the associated KV Cache are strictly limited to a fixed number of the most recent tokens. As new tokens are generated, the oldest tokens are evicted from the cache. This provides a hard upper bound on memory consumption and is essential for processing infinite data streams. It is the core mechanism behind frameworks like StreamingLLM, which enables models trained on finite contexts to handle arbitrarily long sequences by maintaining a cache of recent tokens and a few initial attention sink tokens for stability.

Selective Caching involves identifying and storing only a subset of computed states deemed critical for future steps. A prominent research example is Gist Tokens.

• During an initial processing pass, the model is prompted to identify or generate compact "gist" representations of computationally expensive components (e.g., the output of a large retrieved document).
• These gist tokens are then cached. In subsequent generations, the cached gists are inserted into the prompt, standing in for the full original content.
• This dramatically reduces the token footprint of repeated context, moving the compression cost upstream to a single pass. It is a form of lossy context compression optimized for task performance.

Context caching does not operate in isolation; it is part of a hierarchical memory architecture. The fast, in-memory cache (KV Cache, recent history) sits in front of slower, high-capacity external memory stores.

• Vector Databases: Store long-term semantic memories (e.g., documents, past episodes). The cache holds the most recently retrieved snippets.
• Knowledge Graphs: Store structured facts and relationships. Cached context may include sub-graphs relevant to the current reasoning chain.
• The caching layer provides low-latency access to the "working set" of context, while the external stores act as the backing store for cache misses. This pattern is analogous to CPU cache hierarchies, optimized for the access patterns of LLM agents.

Cache eviction is the process of selectively removing entries from a KV Cache or other context cache to manage finite memory resources. It is governed by a policy that determines which cached data is least valuable to retain.

• Common Policies: Least Recently Used (LRU) discards the oldest accessed data; First-In-First-Out (FIFO) removes the earliest cached data.
• Trigger: Typically occurs when the context window is saturated or a predefined memory limit is reached.
• Engineering Consideration: The eviction policy is a critical design choice that balances memory footprint against the potential need to re-compute evicted context.

Context window optimization is the engineering practice of strategically selecting, ordering, and compressing information to maximize the utility of a model's limited token budget. Caching is one tool within this broader discipline.

• Goal: Achieve the best possible task performance given a fixed token limit.
• Techniques: Includes context caching for reuse, summarization for compression, semantic chunking for retrieval, and intelligent prompt structuring.
• Application: Essential for building cost-effective and low-latency production agentic systems where every token has a direct computational cost.

A context eviction policy is a deterministic rule set that dictates which pieces of context are removed first from a cache when capacity is exhausted. It is a higher-level analog to cache eviction, often applied to summarized conversation history or retrieved documents.

• Examples: LRU (Least Recently Used) for dialogue turns, FIFO (First-In-First-Out) for linear streams, or relevance-based policies that discard the lowest-scoring retrieved chunks.
• Design Factor: Directly impacts an agent's ability to maintain coherent long-term state and avoid catastrophic forgetting of early information.
• Implementation: A key configurable component in Context Management APIs like those in LangChain or LlamaIndex.

Dynamic context refers to an adaptive approach where the content within a model's working window is continuously updated, filtered, or summarized in real-time based on the evolving task. Caching is often used to preserve useful state within this dynamic flow.

• Contrast with Static Context: Unlike a static prompt, dynamic context reacts to new inputs and intermediate outputs.
• Implementation: Involves a loop of context retrieval (from vector stores), caching of critical decisions or state, and eviction of irrelevant details.
• Goal: Maintain a minimal, highly relevant token set to ground the model's reasoning without window saturation.