Context Window

The context window is measured in tokens, not words. A token is roughly 0.75 words in English. A model's maximum context length is a fixed architectural constraint determined during training.

• GPT-4 Turbo: 128k tokens (~96,000 words)
• Claude 3: 200k tokens (~150,000 words)
• Gemini 1.5 Pro: Up to 1M tokens in research previews
• Llama 3: 8k tokens natively

Exceeding this limit requires truncation or chunking strategies, which can break semantic coherence.

Models do not attend to all parts of a long context equally. Research shows a U-shaped attention curve: information at the very beginning and very end of a prompt is recalled most accurately, while content in the middle is frequently overlooked.

• Primacy Bias: Strong recall of initial instructions
• Recency Bias: Strong recall of final statements
• Middle Degradation: Significant performance drop for facts placed between 25% and 75% of the context length

This necessitates strategic prompt engineering, placing critical instructions either at the start or the end of the context.

Transformers are inherently permutation-invariant; they have no built-in sense of sequence order. Positional encodings are vectors added to token embeddings to inject order information.

• Absolute Positional Encoding (APE): Assigns a unique vector to each position (e.g., original Transformer). Limited to the max length seen during training.
• Rotary Position Embedding (RoPE): Encodes relative position by rotating query and key vectors. Enables better length generalization and is used in Llama, Mistral, and PaLM.
• ALiBi: Adds a linear bias to attention scores based on distance. Allows extrapolation to unseen lengths without fine-tuning.

Extending a pre-trained model's context window without full retraining is a major area of research. Key methods include:

• RoPE Scaling: Modifies the rotary frequencies to compress long-range positions into the original training range. Techniques include Linear Scaling, NTK-aware Scaling, and YaRN.
• Position Interpolation: Linearly downscales position indices so that a longer sequence fits within the original context length's positional range.
• Sparse Attention: Uses sliding windows or dilated patterns to reduce the quadratic complexity of full attention, enabling longer sequences (e.g., Longformer, BigBird).

The standard self-attention mechanism has quadratic complexity O(n²) with respect to sequence length n. This creates a hard bottleneck for long context windows.

• KV Cache Growth: The memory required to store Key and Value tensors grows linearly with sequence length. A 1M token context can consume hundreds of GB of VRAM for the cache alone.
• FlashAttention: An IO-aware algorithm that tiles computation to avoid materializing the full NxN attention matrix in slow HBM, reducing memory reads/writes by up to 7.6x.
• Ring Attention: Distributes the long sequence computation across multiple devices in a ring topology, effectively scaling context length linearly with the number of GPUs.

The standard benchmark for long-context retrieval is the Needle In A Haystack (NIAH) test. A specific fact (the 'needle') is placed at a random depth within a long, irrelevant document (the 'haystack'), and the model is queried to retrieve it.

• Pressure Testing: Evaluates recall accuracy across varying context lengths and needle positions.
• RULER Benchmark: An extension by NVIDIA that adds multi-needle, variable tracking, and aggregation tasks to stress-test context utilization beyond simple retrieval.
• Practical Implication: High NIAH scores do not guarantee robust reasoning over the entire context, only retrieval.