Multi-Turn Context

The Conversation History Buffer is the raw, sequential log of all exchanges in a session. It is the primary data structure for multi-turn context.

• Structure: Typically stored as an array of message objects with role (user, assistant, system) and content.
• Challenge: This buffer grows linearly with each turn, directly consuming the model's context window.
• Management: Requires active strategies like context truncation or summarization to prevent context window saturation. The order of messages is critical, as transformers process context sequentially.

The System Prompt defines the assistant's persona, constraints, and behavioral guidelines. It is a persistent, high-priority component of the context.

• Function: Provides grounding and guardrails that apply across all turns (e.g., "You are a helpful coding assistant. Never write insecure code.").
• Placement: Typically inserted at the very beginning of the context window to ensure maximum influence.
• Engineering Consideration: Must be concise. A verbose system prompt permanently reduces the token budget available for the conversation history and user queries.

A Context Compression Engine applies algorithms to reduce the token footprint of the conversation history while attempting to preserve semantic utility.

• Common Techniques:
  • Summarization: Using an LLM to condense past dialogue into a brief abstract.
  • Filtering: Removing tokens deemed irrelevant to the current turn (e.g., greetings, pleasantries).
  • Distillation: Extracting only key facts, decisions, or user preferences.
• Trade-off: Compression risks information loss or introducing hallucinations in the summary. The engine must decide what to compress and when to trigger compression.

Retrieved Context refers to information fetched from an external knowledge source (e.g., vector database, knowledge graph) and injected into the context window to ground the model's responses.

• Mechanism: For a user query, a retrieval system finds the most relevant document chunks via semantic search. These chunks are appended to the prompt.
• Multi-Turn Nuance: In a conversation, retrieval must consider the entire dialogue history to understand the user's intent, not just the latest utterance. This is known as conversational search or query rewriting.
• Token Cost: Retrieved documents consume significant context budget, competing with conversation history.

State & Entity Tracking is the process of explicitly maintaining a structured representation of key information derived from the conversation flow.

• Purpose: To overcome the model's limited working memory and provide deterministic access to facts.
• What is Tracked:
  • Entities: People, places, dates, numbers mentioned.
  • User Preferences: Explicitly stated likes/dislikes or constraints.
  • Task State: Current step in a multi-step process, decisions made, unresolved issues.
• Implementation: Often maintained in a separate data store (a stateful memory) and referenced or injected into the context only when needed, reducing token consumption versus storing the entire raw history.

An Eviction & Prioritization Policy is the rule-based or learned algorithm that decides which parts of the context to keep, compress, or discard when the token limit is approached.

• Core Problem: Context window saturation.
• Common Policies:
  • Least-Recently-Used (LRU): Discard the oldest turns first.
  • Importance Scoring: Use a small model to score the relevance of each past turn to the current dialogue, keeping high-scoring segments.
  • Fixed Schema: Always keep the system prompt and the last N turns, summarizing everything older.
• Goal: Maximize the utility per token within the constrained context window to maintain coherence over long dialogues.

The fixed-size, sequential block of tokens that a transformer-based language model can attend to in a single forward pass. This is the fundamental hardware constraint that multi-turn context management must work within.

• Key Constraint: Defines the absolute upper limit for any conversation history, instructions, and retrieved knowledge combined.
• Unit of Measurement: Typically specified in tokens (e.g., 128K tokens).
• Implication for Agents: The entire operational state—memory, plan, and current dialogue—must fit within this window or be strategically managed.

The process of discarding tokens from a sequence to forcibly fit it within a model's token limit. This is the simplest, often most destructive, method for managing overflowing context.

• Common Strategies: Removing the oldest turns (FIFO), removing middle turns, or removing turns deemed least relevant.
• Primary Risk: Catastrophic forgetting, where the agent loses critical information from early in the conversation, breaking coherence.
• Engineering Use: Often a fallback mechanism when more sophisticated compression fails or is too costly.

A compression technique where a language model generates a concise abstract of the conversation history or document content. This reduces token count while attempting to preserve semantic meaning.

• Agentic Application: Used to condense past dialogue turns into a single, dense summary that is fed back into the context window.
• Trade-off: Introduces inference overhead (cost of the summarization call) and risks the summary model omitting or distorting critical details.
• Example: After 10 dialogue turns, the agent might summarize turns 1-8 into a 3-sentence summary, freeing up tokens for turns 9, 10, and future interaction.

A transformer optimization that stores computed key and value tensors for previous tokens during autoregressive generation. It is the primary technical representation of "context" during model inference.

• Purpose: Eliminates redundant computation for tokens that remain in context, dramatically speeding up sequential token generation.
• Direct Link to Multi-Turn Context: The KV Cache for the entire conversation history must reside in GPU memory. Cache eviction policies are needed to manage this when the context window saturates.
• Engineering Reality: Efficient multi-turn systems must manage the KV Cache's memory footprint as diligently as the prompt text itself.

The process of fetching relevant information from an external memory store (like a vector database) based on the current conversational state, to be injected into the context window.

• Core of RAG for Agents: Enables the agent to maintain a vast long-term memory externally, retrieving only the pertinent pieces (chunks) for the current turn.
• Solves Window Limitation: Shifts the burden from remembering everything in-context to finding and loading the right information on-demand.
• Multi-Turn Nuance: The retrieval query must be updated each turn based on the latest dialogue, making it a dynamic component of context management.

The engineering practice of strategically selecting, ordering, and compressing all information placed into a model's context window to maximize utility per token.

• Key Activities:
  • Priority Ordering: Placing the most critical instructions (system prompt) and recent turns at positions of high attention.
  • Token Budgeting: Allocating fixed token slots for history, tools, retrieved context, and output space.
  • Hybrid Strategies: Combining summarization, retrieval, and selective truncation based on real-time needs.
• Goal: To achieve the highest task performance given the immutable constraint of the token limit.