Context Window

A context window is a fixed-size, contiguous block of tokens. Unlike a computer's RAM, its size (e.g., 128K tokens) is hardcoded during model training and cannot be dynamically expanded at runtime. All input and generated output tokens must fit within this single, unbroken sequence. This constraint necessitates techniques like context chunking and truncation for processing long documents.

Every character, word, and punctuation mark consumed or produced is counted in tokens, not bytes or words. The context budget is shared between:

• Input (Prompt) Tokens: The user's query, instructions, and provided examples.
• Output (Completion) Tokens: The model's generated response. Exceeding the total limit results in an error. This requires careful token limit management, especially for long conversations or document analysis.

The context window's contents are processed via the transformer's self-attention mechanism. Each token can attend to all other tokens within the window, allowing the model to establish contextual relationships. However, this attention has a quadratic computational cost (O(n²)) relative to sequence length, which is the fundamental reason for the fixed context limit. Architectures like sparse attention and sliding window attention were invented to mitigate this cost.

During autoregressive generation, the model maintains a KV Cache (Key-Value Cache) for all previous tokens in the window. This cache stores intermediate computations, allowing the model to generate the next token without re-processing the entire sequence from scratch. Efficient management of this cache is critical for inference speed and handling long contexts, leading to strategies like cache eviction.

Not all parts of the context are equally accessible. Models often exhibit positional bias, where information in the middle of a very long context may be less reliably attended to than information at the beginning or end. When the window is full, context eviction policies (e.g., removing the oldest tokens) must be applied, potentially discarding critical information. This makes context prioritization and summarization key engineering challenges.

The fixed context window is the primary bottleneck for applications requiring long-term memory or processing of extensive documents, such as legal review, long-form conversation, or codebase analysis. This has spurred entire subfields like Retrieval-Augmented Generation (RAG) and memory-augmented networks, which use external databases to circumvent the inherent memory limitation of the transformer architecture itself.

A set of techniques used to reduce the token footprint of information within the context window while preserving semantic utility. This is critical for fitting more relevant data into a fixed token budget.

Key methods include:

• Summarization: Condensing long passages into concise, information-dense representations.
• Redundancy Elimination: Identifying and removing duplicate or highly similar content.
• Selective Pruning: Algorithmically scoring and removing tokens deemed less critical to the current task.

These techniques are foundational for applications like long-document analysis and multi-turn conversational agents.

The strategy of dividing a long document or input stream into smaller, contiguous segments that fit within the model's context limit for sequential or parallel processing.

Common patterns:

• Sequential Processing: Processing chunks in order, often passing a summary or key context forward.
• Hierarchical Processing: Using a smaller model to summarize chunks before feeding summaries to a larger model.
• Map-Reduce: Processing chunks independently (map) and then combining the results (reduce).

This is a fundamental technique for handling inputs that exceed the native context window, such as legal documents or codebases.

The optimization of the Key-Value (KV) Cache, a transformer mechanism that stores computed attention vectors to avoid redundant computation during autoregressive generation.

Critical management operations:

• Cache Eviction: Using algorithms like Least-Recently-Used (LRU) to discard parts of the cache when GPU memory is full during long generations.
• Incremental Encoding: Adding new tokens to an existing cache without recomputing the entire sequence, enabling low-latency streaming.
• Cache Compression: Applying quantization or pruning to the KV cache to reduce its memory footprint.

Efficient KV cache management is directly tied to inference speed and the practical length of generated sequences.

A specific sparse attention pattern where each token can only attend to a fixed number of preceding tokens within a local window. This reduces the computational complexity of self-attention from quadratic to linear with respect to sequence length.

Key implications:

• Enables models to process sequences far longer than their training context window (e.g., 1M+ tokens).
• The effective context is dynamic; as the window slides, the model loses access to information outside the local window unless explicitly managed.
• Architectures like Mistral's and Llama's use this to achieve superior long-context performance.

This is a core architectural technique for scaling context without prohibitive compute costs.

The algorithmic scoring and selective removal of information from the context window to make room for new inputs.

Eviction strategies include:

• Time-based (FIFO): Removing the oldest tokens first.
• Importance-based: Using attention scores or a learned scorer to identify and remove low-utility tokens.
• Structured Eviction: Removing entire turns from a conversation history or specific document sections.

Prioritization often works in tandem, ranking system instructions, recent user queries, and retrieved documents to determine what must be retained. This is essential for maintaining context freshness in long-running sessions.

System architectures designed to handle continuous, unbounded input streams by dynamically managing a rolling context window and KV cache.

Core components:

• Streaming Context: A continuously updated buffer of the most recent tokens, often using a sliding window.
• State Management: Persisting condensed state (e.g., vector summaries) between windows to maintain coherence.
• Low-Latency Inference: Leveraging incremental encoding to add tokens without full forward passes.

This paradigm is critical for real-time applications like live transcription, conversational AI, and monitoring systems, where the input has no predefined end.

The KV Cache is a transformer optimization that stores computed key and value matrices for previous tokens during autoregressive generation. This prevents recomputation, enabling faster inference but consuming GPU memory proportional to the context length.

• Memory Trade-off: Enables efficient generation but memory usage grows linearly with context size.
• Core Mechanism: The KV cache is the technical implementation that makes long-context inference possible without O(n²) recomputation.
• Eviction Policies: When the cache is full, algorithms like Least Recently Used (LRU) determine which past tokens to discard.

Context compression is a set of techniques to reduce the token footprint of information within the context window while preserving semantic utility. It is essential for working with documents longer than the model's limit.

• Techniques Include: Summarization, selective pruning, and redundancy elimination.
• Goal: Maximize information density per token.
• Application: Used in Retrieval-Augmented Generation (RAG) to condense retrieved documents before injection into the context.

Sparse attention is a class of transformer architectures that restrict the full self-attention mechanism to a subset of token pairs. This reduces computational complexity from quadratic (O(n²)) to near-linear, enabling longer context windows.

• Architectural Solution: Addresses the fundamental computational bottleneck of full attention.
• Patterns: Includes sliding window attention, where a token only attends to a fixed local window of preceding tokens.
• Examples: Models like Longformer and BigBird use sparse attention patterns to process documents of up to thousands of tokens.

Context chunking is the strategy of dividing a long document into smaller segments that fit within the model's context limit for sequential or parallel processing. It's a fundamental technique for long-document analysis.

• Simple but Effective: The most common method for processing long texts with fixed-context models.
• Challenges: Requires strategies to maintain coherence and state across chunks.
• Use Case: Essential for tasks like document summarization, Q&A over manuals, and codebase analysis.

Positional encoding is the method of injecting information about the absolute or relative order of tokens into a transformer's input embeddings. Since self-attention is permutation-invariant, this is critical for the model to understand sequence position within the context window.

• Absolute vs. Relative: Early models (e.g., GPT-2) used absolute sinusoidal encodings; newer architectures often use more flexible relative positional encodings.
• Context Length Extrapolation: A major research area is designing positional encodings that generalize to sequences longer than those seen in training.
• Rotary Positional Embedding (RoPE): A common technique in modern LLMs like LLaMA that provides relative positional information.