Context Window

A context window's size is measured in tokens, not words or characters. A token is a subword unit generated by a tokenizer like Byte-Pair Encoding (BPE). A single English word averages ~1.3 tokens, but complex or agglutinative words can fragment into many more. The model's attention mechanism computes pairwise relationships across all tokens in this window, making the computational cost scale quadratically with window size.

• GPT-4 Turbo: 128,000 tokens (~96,000 words)
• Claude 3 Opus: 200,000 tokens (~150,000 words)
• Gemini 1.5 Pro: Up to 1,000,000 tokens in research preview
• Mistral Large: 32,000 tokens

A well-documented failure mode where models exhibit degraded recall for information positioned in the middle of a long context window. Performance follows a U-shaped curve: the model best remembers the beginning (primacy bias) and the end (recency bias) of the prompt, while the central portion suffers significant attention dropout.

• This is not a token limit issue but an attention distribution failure
• Mitigation strategies include reordering documents by relevance and placing critical instructions at the extremes
• Evaluated using the Needle-in-a-Haystack benchmark, where a random fact is inserted at varying depths

Since the Transformer architecture is permutation-invariant, it requires a mechanism to inject sequence order into token representations. Positional encodings assign each token a unique vector based on its absolute or relative position in the context window.

• Absolute Positional Encoding: Uses sinusoidal functions or learned embeddings tied to fixed indices (original Transformer)
• Rotary Position Embedding (RoPE): Encodes relative position by rotating query and key vectors, enabling better length extrapolation (used in LLaMA, Mistral, PaLM)
• ALiBi: Adds a linear bias to attention scores based on token distance, penalizing far-away tokens without learned embeddings

The mechanism that controls which tokens can attend to which other tokens. In autoregressive (causal) language models, a triangular mask is applied to the attention matrix, preventing each token from attending to future tokens. This enforces the left-to-right generation constraint.

• Causal Mask: Token at position i can only attend to positions 0 through i
• Padding Mask: Ignores filler tokens in batched inputs of unequal length
• Custom Masks: Can enforce document boundaries, speaker turns, or other structural constraints within the window
• Masking is implemented by adding negative infinity to prohibited attention scores before the softmax

Methods to increase the effective context length of a pre-trained model beyond its original training window without full retraining. These techniques modify the attention mechanism or positional encoding to generalize to longer sequences.

• Position Interpolation: Linearly scales down all position indices to fit within the original trained range, then fine-tunes on longer sequences
• NTK-Aware Scaling: Modifies the base frequency of RoPE to prevent high-frequency information loss during interpolation
• YaRN (Yet another RoPE extensioN): Combines NTK-aware interpolation with a temperature factor on attention logits to maintain perplexity at extended lengths
• Self-Extend: Prompts the model to group tokens into chunks, using the original window size for local attention and a larger stride for global context

During autoregressive generation, the Key and Value tensors computed for all previous tokens are cached in GPU memory to avoid redundant recomputation. The size of this KV-cache grows linearly with the context window and batch size, becoming the primary memory bottleneck for long-context inference.

• Memory formula: 2 * num_layers * hidden_dim * context_length * batch_size * precision_bytes
• Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) reduce KV-cache size by sharing key/value heads across query heads
• PagedAttention (vLLM) manages KV-cache in non-contiguous memory blocks, analogous to virtual memory paging, to minimize fragmentation
• Sliding Window Attention (Mistral) restricts attention to a fixed local window, capping KV-cache growth regardless of total context length

The core architectural component that allows a model to dynamically weigh the importance of different tokens within the context window. Rather than processing all input equally, the attention mechanism computes a weighted sum of values, enabling the model to 'attend' to semantically relevant words regardless of their distance in the sequence. The self-attention variant, central to the Transformer architecture, calculates pairwise relationships between every token in the window, creating a rich contextual representation. Computational complexity scales quadratically with context length, making this the primary bottleneck for extending context windows.

The preprocessing step that converts raw text into discrete units—tokens—that a model can ingest. The choice of tokenizer directly impacts effective context window utilization. Common approaches include:

• Byte-Pair Encoding (BPE): Iteratively merges frequent character pairs to build a subword vocabulary
• WordPiece: Similar to BPE but selects merges based on likelihood rather than frequency
• SentencePiece: A language-agnostic tokenizer treating input as a raw byte stream Inefficient tokenization can waste context window capacity by fragmenting words into excessive subword units.

A mathematical scheme that injects information about token order into the model, since the attention mechanism itself is permutation-invariant—it sees the context window as an unordered set. Without positional encoding, 'The dog bit the man' and 'The man bit the dog' would be indistinguishable. Common approaches include:

• Sinusoidal encodings: Fixed functions of position and dimension used in the original Transformer
• Learned positional embeddings: Trainable vectors assigned to each absolute position
• Rotary Position Embedding (RoPE): Encodes relative position by rotating query and key vectors, enabling better length extrapolation beyond the training context window

A memory optimization technique that stores pre-computed Key and Value tensors from previous generation steps to avoid redundant computation. During autoregressive decoding, each new token only needs to attend to all prior tokens—the model doesn't recompute attention for the entire sequence. The KV cache grows linearly with sequence length and is the primary consumer of GPU memory during inference. For a context window of size N, the cache stores N key-value pairs per layer, making it the binding constraint on maximum context length for a given hardware configuration.

An attention pattern variant that restricts each token's receptive field to a fixed-size local neighborhood rather than the full context window. For example, with a window size of 512, each token attends only to the 512 tokens preceding it. This reduces the quadratic complexity of full attention to linear, enabling dramatically longer sequences. Mistral and Longformer models employ this technique. The trade-off is that information outside the sliding window requires multiple layers to propagate, potentially losing long-range dependencies that a full-context model would capture directly.

A benchmark methodology that tests a model's ability to retrieve a specific piece of information—the needle—placed at varying positions within a long context window filled with distractor text—the haystack. The test reveals whether a model's effective context utilization degrades with length. Key findings across models:

• Many models exhibit a 'lost in the middle' phenomenon, where information in the central portion of the context window is poorly retrieved
• Performance often remains strong at the very beginning and end of the window
• This benchmark has become the standard for evaluating long-context model claims