Context Window

The context window is measured in tokens, not words or characters. A token is a sub-word unit generated by the model's tokenizer, typically representing ~4 characters or 0.75 words in English text. For legal documents, this distinction is critical: a single complex statutory citation like '42 U.S.C. § 1983' may consume 6-8 tokens. Context windows range from 4K tokens in legacy models to 128K, 200K, or even 1M tokens in frontier architectures. A 128K-token window can hold approximately 96,000 words—roughly the length of a 300-page legal brief. The window encompasses the system prompt, user message, conversation history, and the model's generated response combined.

The standard self-attention mechanism scales quadratically with context length: O(n²) in both computation and memory, where n is the number of tokens. A 128K-token window requires processing over 16 billion attention pairs per layer. This creates a fundamental tension between context capacity and inference cost. Architectural innovations addressing this include FlashAttention (IO-aware exact attention), ring attention (distributed computation across devices), and sparse attention patterns (Sliding Window Attention, Dilated Attention) that reduce complexity to O(n log n) or O(n). For legal deployments, this directly impacts the cost-per-query and latency of processing full case dockets.

During autoregressive generation, the model stores computed Key-Value (KV) pairs for all previous tokens to avoid recomputation—this is the KV-cache. For long legal prompts, the KV-cache can consume gigabytes of GPU memory, becoming the primary bottleneck. Prompt caching allows reuse of the KV-cache for repeated system prompts or static document prefixes across multiple queries, dramatically reducing latency and cost. Providers implement context caching where identical prefix tokens are computed once and shared across requests. For legal RAG systems, structuring prompts with a static, cacheable system prompt and document preamble can yield 50-80% latency reductions on repeated queries against the same corpus.

The process of converting raw legal text into discrete tokens—the atomic units a language model processes. A context window's capacity is measured in tokens, not words or characters. For legal text, tokenization efficiency varies significantly: a dense contract clause may consume more tokens than a conversational sentence of the same character length. Subword tokenization methods like Byte-Pair Encoding (BPE) break rare legal terms (e.g., res judicata) into multiple tokens, while common words map to single tokens. Understanding tokenization is essential for accurately calculating whether a multi-document corpus fits within a model's context budget.

The mathematical mechanism that injects sequence order information into token embeddings, allowing a transformer model to distinguish 'The plaintiff sued the defendant' from 'The defendant sued the plaintiff'. Without positional encoding, the self-attention mechanism would treat input tokens as an unordered bag. Early transformers used sinusoidal positional encodings; modern architectures like Rotary Position Embedding (RoPE) enable better extrapolation to sequence lengths beyond those seen during training. This directly impacts a model's ability to reason over long legal documents where clause order determines meaning.

The core computational operation that allows a model to weigh the relevance of every token in the context window when processing each individual token. Self-attention computes pairwise similarity scores across the entire input sequence. The computational complexity scales quadratically with context length—doubling the window quadruples the compute. This is the primary bottleneck limiting context window expansion. For legal reasoning, attention enables a model to connect a reference in Section 12 of a contract to its definition in Section 1, but only if both fit within the window.

The practice of packing as much relevant legal material as possible into a single prompt to maximize the model's available information. This includes concatenating multiple contracts, statutes, and case excerpts. Effective context stuffing requires careful prioritization: placing the most critical documents at the beginning and end of the prompt, as models often exhibit a 'lost in the middle' phenomenon where information in the central portion of a long context receives less attention. Poor stuffing strategies degrade retrieval accuracy even when documents technically fit within the window.

A well-documented failure mode where language models exhibit significantly degraded recall for information positioned in the middle of a long context window. Performance is strongest at the very beginning and very end of the input. For legal applications, this means a critical clause buried in the middle of a 100-page contract may be effectively invisible to the model, even though it is technically within the context window. Mitigation strategies include reordering documents by relevance, using retrieval-augmented generation (RAG) to surface key passages, or employing structured summarization of intermediate sections.

An architectural optimization that restricts each token's attention to a fixed-size local neighborhood rather than the entire context. This reduces the quadratic complexity of full attention to linear, enabling much longer effective context windows. Models like Longformer and Mistral employ this technique. For legal document processing, sliding window attention allows a model to process a 500-page merger agreement by attending to local sections (e.g., individual clauses) while relying on global tokens or specialized memory mechanisms to capture cross-document relationships like defined terms and cross-references.