Context Window

The context window is measured in tokens, not words or characters. A token is roughly 0.75 words in English. A model with a 128K context window can process approximately 96,000 words in a single forward pass. Exceeding this limit requires chunking strategies or summarization, as tokens beyond the window are simply ignored by the attention mechanism.

The computational cost of self-attention scales quadratically with sequence length. Doubling the context window quadruples the memory and compute required for the attention score matrix. This is why extremely long context windows require architectural innovations like sparse attention, FlashAttention, or state-space models to remain computationally feasible.

Since self-attention is permutation-invariant, models rely on positional encodings to understand token order. Context window extensions require positional encoding schemes that can generalize beyond their training length. Rotary Position Embedding (RoPE) with interpolation techniques like NTK-aware scaling allows models to extrapolate to longer sequences than those seen during training.

During autoregressive generation, the Key-Value (KV) cache stores attention states for all previous tokens to avoid recomputation. The cache size grows linearly with context length and batch size. For a 70B parameter model with a 128K context, the KV cache can consume over 100GB of GPU memory, making it the dominant bottleneck for long-context inference.

A model's advertised context window often exceeds its effective context. Needle-in-a-Haystack evaluations reveal that many models fail to reliably retrieve information distributed across the full advertised length. True long-context capability requires both architectural support and training on long sequences, not just positional interpolation at inference time.

The fundamental computational constraint that defines why context windows cannot be arbitrarily large. In standard self-attention, both memory and time complexity scale quadratically with sequence length (O(n²)). Doubling the context window quadruples the compute required, making naive scaling to 1M+ tokens prohibitive without architectural innovations like sparse attention or state space models.

The mechanism that injects sequence order information into the permutation-invariant self-attention operation. Without it, the model treats input as a bag of tokens. Modern approaches like Rotary Position Embedding (RoPE) encode relative position directly into the Query-Key dot product, enabling superior length extrapolation beyond the trained context window.

An inference optimization that stores previously computed Key and Value vectors to avoid redundant recomputation during autoregressive generation. The cache size grows linearly with both batch size and sequence length, making it the primary memory bottleneck for long-context inference. Techniques like Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) reduce this footprint.

An IO-aware exact attention algorithm that tames the quadratic bottleneck by minimizing HBM reads/writes. It fuses operations in GPU SRAM using tiling and recomputation, achieving 2-4x speedups. This hardware-level optimization is critical for training models with large context windows without sacrificing the exactness of full attention.

A class of techniques that reduces complexity from O(n²) to O(n log n) or O(n) by computing only a subset of the attention matrix. Patterns include:

• Sliding Window: Attend only to local neighbors
• Dilated Sliding: Skip intervals for wider receptive field
• Global Tokens: Designate special tokens that attend everywhere This enables processing of documents far exceeding the dense context window.

A triangular mask that prevents tokens from attending to future positions, enforcing the autoregressive property in decoder-only models. This mask defines the effective context for each token: position i can only see positions 0 through i. During training, this mask enables parallel computation across the full sequence while maintaining causal integrity.