KV Cache State

During autoregressive generation, a transformer model processes one token at a time. For each new token, the self-attention mechanism computes a weighted sum over all previous tokens in the sequence. The keys (K) and values (V) from these previous tokens are identical for each new generation step. The KV Cache stores these computed K and V tensors from prior tokens, preventing their redundant recalculation. This is the core optimization: compute K and V once, cache them, and reuse them for all future attention computations in that sequence.

• Without KV Cache: Each new token generation requires recomputing K and V for all preceding tokens, leading to O(n²) computational complexity in sequence length.
• With KV Cache: K and V are computed once and retrieved from memory, reducing the complexity to O(n) for the attention operation.

The KV Cache is not a single blob of data but a structured, multi-dimensional tensor for each layer of the transformer. For a model with h attention heads, a context length of n, and a head dimension of d, the cached tensors have the shape [n, h, d] for both keys and values. This structure is maintained per layer.

• Batch Inference: In continuous batching systems, the cache is managed across multiple concurrent sequences (a batch). The effective shape becomes [batch_size, n, h, d], requiring sophisticated memory management to handle sequences of different lengths within the same batch.
• Memory Footprint: The total size is 2 * layers * batch_size * n * h * d * bytes_per_param. For a large model (e.g., 70B parameters), long contexts (n=128k), and a large batch, this can consume tens to hundreds of gigabytes of GPU memory.

When the sequence length exceeds available memory, cache eviction policies determine which parts of the history to discard or compress. Common strategies include:

• Window-based Attention: Only cache the most recent k tokens (a sliding window), discarding older ones. This assumes distant tokens have minimal influence on the current generation.
• StreamingLLM-style Retention: Maintain the first few tokens (attention sinks) and a sliding window of recent tokens to stabilize attention scores for extremely long sequences.
• Quantization: Apply post-training quantization (e.g., to FP8 or INT4) to the cached K and V tensors, significantly reducing memory usage with a marginal accuracy trade-off.
• Selective Caching: For architectures like Mixture of Experts (MoE), only cache the K/V for activated experts, not all parameters.

In production serving systems like vLLM or TGI, continuous batching dynamically groups sequences of different lengths into a single computational batch. The KV Cache state for each sequence is managed independently but must be efficiently packed into contiguous GPU memory.

• PagedAttention: This algorithm, used in vLLM, manages the KV Cache in fixed-size blocks (pages), analogous to virtual memory. Each sequence owns a set of non-contiguous blocks, allowing for:
  • Zero waste from internal fragmentation.
  • Efficient sharing of cache blocks between sequences in speculative decoding.
  • Simple memory allocation and deallocation as sequences start and finish.
• The scheduler must track the life cycle of each sequence's cache, freeing it immediately upon completion to maximize GPU memory utilization.

The KV Cache must be invalidated or updated when the underlying model state changes, ensuring state consistency.

• During Fine-tuning/LoRA Updates: If model weights are updated via online fine-tuning or a LoRA adapter is swapped, the existing KV Cache becomes stale because the K and V projections are derived from outdated weights. The entire cache must be flushed or recomputed.
• Multi-Turn Dialog: In an agentic context, a user's instruction to "forget what I just said" requires selective invalidation of the cache segments associated with that part of the conversation history.
• Dynamic System Prompts: If the agent's system prompt or instructions change mid-session, the K/V for the initial prompt tokens may need recomputation, as they are conditioned on the new instructions.

For agentic observability, tracking KV Cache state provides crucial performance and health metrics.

• Key Telemetry Signals:
  • Cache Utilization Percentage: (Current Sequence Length / Max Context Length) * 100. Alerts when approaching limit.
  • Cache Memory Allocated: Total GPU memory consumed by KV Cache across all active sequences.
  • Cache Miss Rate: Tracks instances where needed K/V are not in cache (indicative of a bug or invalidated state).
  • Eviction Count: Number of tokens/blocks evicted due to memory pressure.
• Integration with Distributed Tracing: Cache operations (allocation, eviction) can be emitted as spans, linking inference latency spikes directly to cache management overhead. This is vital for debugging performance issues in production agent deployments.

A complete, point-in-time capture of an autonomous agent's internal variables, memory contents, and operational status. This is the foundational unit for debugging, analysis, and recovery.

• Primary Use: Provides a frozen view of the agent for post-mortem analysis or as a recovery point.
• Contrast with KV Cache: While a snapshot captures the entire agent state, the KV Cache is a specific, performance-critical subset focused on transformer layer computations.

The software component responsible for durably storing and retrieving an agent's state from non-volatile storage (e.g., disk, database). This ensures state survival across process restarts or hardware failures.

• Critical for Recovery: Enables agents to resume long-running tasks after interruptions.
• Relation to KV Cache: In some architectures, the KV Cache may be managed or offloaded by this layer when memory pressure is high, trading latency for resilience.

The process of periodically saving an agent's complete operational state to stable storage. This creates known-good recovery points to resume execution after a failure.

• Key Mechanism for Fault Tolerance: Essential for long-lived or critical agents.
• Performance Trade-off: Involves a cost (I/O, latency) to serialize state. For LLMs, checkpointing a large KV Cache can be expensive, influencing the checkpoint interval.

The process of reconstructing an agent's full, operational in-memory state from a persisted snapshot or checkpoint. This is the reverse of checkpointing, turning stored data back into a live, executable agent.

• Booting from a Save Point: Allows an agent to skip initialization and continue from a previous state.
• KV Cache Implication: Rehydrating a pre-computed KV Cache can dramatically speed up the resumption of LLM inference for a specific conversation, avoiding recomputation of initial tokens.

An agent's active operational data held in volatile RAM for fast access during execution. This includes conversation context, tool call results, intermediate reasoning, and the KV Cache.

• Performance-Critical: Directly determines agent responsiveness and throughput.
• Volatile Nature: Lost on process termination unless persisted. The KV Cache is a prime example of high-value, performance-critical in-memory state that is often recomputed rather than persisted due to its size.

A rule-based algorithm that determines which parts of an agent's in-memory state to remove or offload when system resource limits (like memory) are reached. Common policies include Least Recently Used (LRU) or Least Frequently Used (LFU).

• Manages Resource Constraints: Essential for scaling agents with unbounded context.
• Directly Governs KV Cache: In LLM serving systems, the eviction policy decides which past conversation contexts (and their associated KV Caches) to discard first when memory is full, directly impacting performance for older sessions.