Legal Sequence Length

The context window defines the maximum span of text a model can 'see' at once, measured in tokens. In legal AI, this is the single most critical architectural constraint. A typical commercial lease spans 15,000–50,000 tokens, while a complex merger agreement can exceed 200,000 tokens. If a model's context window is too small, it must process documents in chunks, losing cross-references and the ability to reason about obligations defined in Section 2.1 that are triggered by conditions in Section 14.3. Modern legal models target context windows of 128K to 1M tokens to handle entire case files in a single pass.

The fundamental bottleneck of the Transformer architecture is that self-attention scales quadratically with sequence length—O(n²). Doubling the sequence length quadruples the compute and memory required. For a 100,000-token legal document, a standard attention mechanism computes a 100,000 × 100,000 attention matrix, consuming tens of gigabytes of GPU memory. This is why innovations like FlashAttention, Ring Attention, and sparse attention patterns are essential for legal AI. Without them, long-document processing is either computationally infeasible or prohibitively expensive for production deployment.

To understand a legal document, a model must know where each token sits in the sequence. Positional encodings inject this order information. Early methods like sinusoidal encodings degrade on sequences longer than those seen during training. Modern legal models use Rotary Position Embeddings (RoPE) with extended interpolation techniques such as NTK-aware scaling or YaRN to generalize to lengths far beyond their pre-training maximum. Without robust positional encoding, a model cannot distinguish between a clause in the definitions section and an operative provision, undermining legal reasoning.

During autoregressive generation, a model stores the Key and Value tensors for every previous token in a KV-cache to avoid recomputing them. For a 70-billion-parameter model processing a 100,000-token legal brief, the KV-cache alone can consume over 50 GB of GPU memory. This memory pressure is the primary bottleneck for serving long-context legal models. Techniques like Multi-Query Attention (MQA), Grouped-Query Attention (GQA), and KV-cache quantization are deployed to shrink this footprint, enabling practical inference on lengthy legal documents without requiring exotic hardware.

A fundamental architectural decision is whether to chunk long documents into smaller segments or process them natively. Chunking with overlap and a retrieval step is computationally cheaper but breaks cross-reference reasoning—a model cannot connect a liability cap on page 3 with an indemnification clause on page 47. Native long-context processing preserves these dependencies but demands the quadratic-complexity solutions described above. The state-of-the-art for legal AI is moving toward native long-context models with architectures like Mamba-2 or linear attention variants that offer sub-quadratic scaling.

The standard benchmark for long-context legal models is the Needle-in-a-Haystack (NIAH) test. A specific fact—the 'needle'—is placed at a random depth within a long legal document—the 'haystack'—and the model is queried to retrieve it. A model with a 128K context window should achieve near-perfect recall regardless of the needle's position. Failure at the beginning (the 'lost-in-the-middle' problem) or at extreme lengths indicates architectural weaknesses. Legal AI systems must pass NIAH at their full rated context length to be trusted for multi-document reasoning.

An IO-aware exact attention algorithm that dramatically reduces the memory footprint of the self-attention mechanism. By tiling the attention matrix and recomputing softmax statistics in SRAM rather than storing the full matrix in high-bandwidth memory, FlashAttention achieves up to 7.6x speedup on long sequences. This innovation is what makes pre-training on lengthy legal documents economically feasible, as the quadratic memory cost of standard attention would otherwise prohibit processing entire contracts or judicial opinions in a single pass.

During autoregressive decoding, the key and value tensors from previous tokens are cached to avoid redundant recomputation. For long legal documents, this KV cache grows linearly with sequence length and can exceed GPU memory. Advanced techniques include:

• Multi-Query Attention (MQA): Shares a single key-value head across all query heads
• Grouped-Query Attention (GQA): Balances memory savings with quality by grouping query heads
• PagedAttention: Manages KV cache in non-contiguous memory blocks, reducing fragmentation

Techniques for extending a pre-trained model's sequence length beyond its original training configuration without full retraining. Key methods include:

• Position Interpolation: Linearly rescales position indices to fit within the original range
• NTK-aware scaling: Applies Neural Tangent Kernel theory to adjust RoPE base frequencies non-uniformly
• YaRN: Combines NTK scaling with a temperature adjustment to preserve high-frequency information These methods are essential for adapting general-purpose models to the extreme length requirements of legal documents.

Attention mechanisms that restrict each token to attending to only a subset of other tokens, reducing the quadratic O(n²) complexity to O(n log n) or O(n). Patterns relevant to legal text include:

• Sliding Window: Each token attends to a fixed local neighborhood, capturing clause-level context
• Global Tokens: Designated tokens attend to the entire sequence, preserving document-level understanding
• Dilated Sliding Window: Expands the receptive field exponentially, useful for hierarchical legal document structures These patterns trade perfect recall for computational tractability on extremely long texts.