Context Drift

As the conversation history grows, the model's self-attention mechanism must distribute its weights across an increasingly large number of tokens. This causes the initial instructions and early key facts to receive exponentially less attention weight, a phenomenon directly linked to the quadratic complexity of the transformer architecture. The model begins to treat the most recent user message as the primary signal, effectively forgetting the original system prompt or the first user query that established the task's constraints.

Unlike a hard context window overflow, drift is a subtle, continuous process. The model does not abruptly forget; it gradually shifts its semantic focus. A conversation that begins with a detailed financial analysis may, over 20 turns, drift toward general economic commentary. This occurs because each new response adds tokens that slightly alter the latent semantic centroid of the dialogue, pulling subsequent token predictions away from the original domain without any explicit topic change command.

A critical failure mode of context drift is the model's loss of fidelity to previously established facts. An entity's name, a specific numerical value, or a constraint set in turn 1 may be contradicted by turn 15. This is not a hallucination in the traditional sense but a retrieval failure within the context window. The model's internal representation of the fact becomes 'crowded out' by newer, irrelevant tokens, leading it to generate a plausible but inconsistent continuation based on its parametric knowledge rather than the dialogue state.

Sophisticated prompts often use a hierarchy of instructions: a high-priority system prompt, a task description, and few-shot examples. Context drift causes this hierarchy to flatten. The model begins to treat all prior text with equal weight, causing low-priority conversational filler (e.g., 'thanks, that's helpful') to dilute the high-priority constraints. The model may eventually ignore the system prompt's directive to respond in JSON format or to refuse certain topics, as those instructions are buried deep in the history.

Context drift is exacerbated by the well-documented 'lost-in-the-middle' phenomenon, where models attend most strongly to the beginning and end of their context window. As a conversation grows, the critical middle turns—which often contain the core task parameters and intermediate reasoning steps—fall into an attention trough. The model becomes disproportionately influenced by the most recent exchange and the initial system prompt, while the connective reasoning that links them fades, causing a disjointed and incoherent conversational arc.

The primary defense against context drift is periodic re-anchoring. This involves programmatically re-injecting a compressed version of the original intent and key constraints into the most recent part of the conversation history. Techniques include:

• Summarization triggers: Automatically summarizing the conversation every N turns and prepending it to the next prompt.
• Structured state tracking: Maintaining a separate JSON object of critical facts (names, numbers, decisions) that is injected into the system prompt on every turn.
• Intent re-classification: Running the original query and the last user message through an intent classifier to detect deviation and trigger a corrective re-prompt.