Context Window

The context window's capacity is measured in tokens, not words or characters. A token is an atomic unit from the model's vocabulary, typically representing ~0.75 words in English. The window size is a hard architectural limit set during pre-training and cannot be exceeded at inference time without specialized extrapolation techniques.

• GPT-4 Turbo: 128K tokens
• Claude 3 Opus: 200K tokens
• Gemini 1.5 Pro: Up to 1M tokens

Exceeding the limit triggers a context length exceeded error, requiring truncation or compression strategies.

The standard self-attention mechanism scales with O(n²) computational complexity relative to sequence length n. Doubling the context window quadruples the attention computation and memory requirements.

This quadratic scaling is why long-context models require optimizations like:

• FlashAttention: Minimizes HBM reads/writes via tiling
• Ring Attention: Distributes computation across devices
• Sparse Attention: Limits attention to local or selected tokens

Without these, serving long-context models becomes prohibitively expensive in GPU memory and latency.

Transformers are permutation-invariant by design—they have no inherent sense of token order. The context window's ability to understand sequence depends entirely on positional encodings injected into token embeddings.

Key encoding methods:

• RoPE (Rotary Position Embedding): Encodes relative positions via rotation matrices, enabling superior length extrapolation
• ALiBi: Applies linear distance-based biases to attention scores, no learned embeddings required
• Learned Absolute: Fixed-size lookup table, cannot generalize beyond trained length

The choice of encoding directly determines whether a model can handle sequences longer than its training window.

LLMs exhibit a well-documented U-shaped attention curve: information at the beginning (primacy bias) and end (recency bias) of the context window is retrieved with high accuracy, while content in the middle suffers significant degradation.

• Multi-document QA accuracy can drop by 20-30% for middle-positioned documents
• This effect intensifies as context length increases
• Mitigation strategies include re-ranking to place critical chunks at window boundaries and contextual compression to reduce filler content

The phenomenon is a fundamental limitation of current attention architectures, not a training artifact.

During autoregressive generation, the Key-Value (KV) Cache stores pre-computed attention tensors for all previous tokens to avoid recomputation. For long contexts, this cache dominates GPU memory consumption.

Memory formula for a single layer:

codeCopy
2 × batch_size × num_heads × seq_length × head_dim × precision_bytes

Optimization techniques:

• Grouped-Query Attention (GQA): Shares KV heads across query heads, reducing cache size by 4-8×
• Multi-Query Attention (MQA): Single KV head for all queries, maximum compression
• Prefix Caching: Reuses KV-Cache for shared prompt prefixes across requests

Without these, a 128K context can consume tens of gigabytes of GPU memory for the cache alone.

In Retrieval-Augmented Generation (RAG) architectures, the context window serves as the fusion point where retrieved external knowledge meets the model's parametric knowledge. The window's finite capacity creates a zero-sum allocation problem: every retrieved chunk consumes tokens that could be used for instructions, conversation history, or generation output.

Strategic allocation patterns:

• Instruction Prefix: 5-15% of window for system prompts and task definitions
• Retrieved Context: 60-80% for grounding documents
• Conversation History: 10-20% for multi-turn coherence
• Output Buffer: Reserved tokens for generation

Effective RAG design requires re-ranking to prioritize high-value chunks and contextual compression to maximize information density per token.