Context Window

The context window is measured in tokens, not words or characters. A token is roughly 0.75 words in English, meaning a 128k token window can hold approximately 96,000 words—equivalent to a 300-page novel. This capacity defines the absolute ceiling for how much retrieved context, conversation history, and instruction text can be passed to the model simultaneously. Exceeding this limit triggers truncation errors or requires chunking strategies.

The self-attention mechanism scales quadratically with context length—doubling the window quadruples the compute cost. This creates a fundamental tension between context size and inference speed. Models use techniques like Grouped-Query Attention (GQA) and FlashAttention to mitigate this, but the Lost-in-the-Middle phenomenon persists: models attend most strongly to the beginning and end of the context window, often missing critical information placed in the middle.

The theoretical context window is the model's advertised maximum token limit. The effective context window is the span within which the model actually maintains high recall accuracy. Research shows that even in models with 128k token windows, retrieval accuracy degrades significantly beyond ~73k tokens for factual recall tasks. This gap drives architectural decisions around chunk sizing—chunks must be small enough to fit within the high-accuracy zone while carrying sufficient semantic context.

The context window is distinct from the model's training data cutoff. The window governs how much input the model can process at inference time, while the cutoff determines what knowledge was available during pre-training. A model with a 1M token window but a January 2024 cutoff can ingest a massive document but cannot know about events after that date unless provided via in-context learning within the window itself.

Some architectures employ sliding window attention to extend effective context without quadratic cost. Instead of attending to all tokens, each token attends only to a fixed-size neighborhood (e.g., 4,096 tokens) plus a few global tokens. This creates a receptive field that grows linearly with layers, allowing models like Mistral to process long sequences efficiently. However, this trades off long-range dependency capture for computational tractability.

Modern models achieve large context windows through position interpolation techniques like RoPE scaling. Rather than training from scratch on long sequences, models fine-tune with rescaled position encodings—compressing the original positional space into the extended range. Methods include NTK-aware scaling and YaRN, which adjust the rotary base frequency to preserve high-frequency information while extending low-frequency range. This enables 4x-8x window expansion with minimal fine-tuning.