Context Window

The context window is defined by a strict token limit (e.g., 128k, 1M tokens), representing the maximum span of text the model can attend to at once. This is a hard architectural constraint, not a soft guideline. Exceeding this limit triggers context window truncation, where the oldest tokens are forcibly discarded. This process often severs the system prompt or initial instructions, leading to catastrophic forgetting of the agent's persona and behavioral constraints.

During autoregressive generation, the model recomputes attention for all previous tokens unless a KV-Cache is utilized. This cache stores the Key and Value tensors of processed tokens, avoiding redundant computation. However, the KV-cache grows linearly with the context window, consuming significant GPU VRAM. Techniques like Prompt Caching store embeddings for a static system prefix, while Contextual Compression extracts only relevant snippets to minimize the cache footprint and reduce time-to-first-token latency.

A Causal Attention Mask is a triangular matrix applied during self-attention to enforce autoregressive generation. It prevents the model from attending to future tokens, ensuring the prediction for token t depends only on tokens 0 through t-1. This is distinct from a general Attention Mask, which simply hides padding tokens. The causal mask is the fundamental mechanism that maintains temporal coherence in a dialogue, preventing the model from cheating by looking ahead in the conversation.

The context window is a primary attack surface. Context Poisoning occurs when malicious data is injected into the conversation history or retrieval source to manipulate outputs. A Prompt Injection Boundary is the logical delimiter separating untrusted user input from trusted developer instructions. Without strict formatting via a Chat Template (like ChatML), the model cannot distinguish between a user's malicious command and the system's core directives, leading to security breaches.

Over long multi-turn interactions, Context Drift occurs as the conversation gradually deviates from the original intent, causing the model to lose focus. A more severe failure state is Context Collapse, where the model loses the distinction between different conversational threads or temporal states, flattening the dialogue into an incoherent prompt. This often results from poor Dialogue State Tracking (DST) or exceeding the window's effective reasoning capacity, not just its token limit.

The process of discarding the oldest tokens from a context window when the cumulative token count exceeds the model's maximum limit. This is a lossy operation that can sever the model's connection to its initial instructions.

• FIFO Strategy: First-in, first-out removal of tokens from the head of the sequence.
• Risk: System prompts and early dialogue turns are the first to be evicted, causing personality drift.
• Alternative: Sliding window attention or contextual compression to summarize older turns before truncation.

The process of extracting only the relevant snippets from a long context or retrieved document to fit within the model's maximum token limit without losing fidelity. This is distinct from naive truncation.

• Approach: Use a smaller, fast model to score and filter sentences based on their relevance to the query.
• Contrast with RAG: Compression operates on already-retrieved context, while RAG retrieves from an external index.
• Benefit: Preserves the semantic density of the prompt by removing filler text while retaining critical facts.

A triangular matrix that ensures autoregressive generation by masking all future token positions during self-attention. Each token can only attend to itself and preceding tokens, never to tokens that have not yet been generated.

• Structure: An upper-triangular matrix of negative infinity values applied before the softmax operation.
• Purpose: Prevents information leakage from the future, enforcing the left-to-right generation constraint.
• Contrast: Bidirectional attention (used in encoders like BERT) allows every token to attend to every other token simultaneously.