Longformer

The core innovation replacing full self-attention. Each token attends only to a fixed-size window of w neighboring tokens on each side, reducing complexity from O(n²) to O(n*w).

• Fixed Context Radius: Typically 512 tokens on each side
• Stacked Layers: Receptive field grows with depth, eventually covering the entire document
• Legal Relevance: Captures local syntactic structures and clause boundaries without quadratic cost

Designated tokens—such as [CLS] for classification or question tokens in QA tasks—receive global attention, attending to all other tokens in the sequence.

• Task-Specific Flexibility: Configure which tokens operate globally based on the downstream task
• Information Bottleneck: Global tokens aggregate document-level context for sequence-level predictions
• Legal Application: Enables a [CLS] token to synthesize an entire 50-page contract for clause classification

To increase the receptive field without increasing computation, the sliding window can be dilated—introducing gaps between attended tokens.

• Exponential Growth: Dilation doubles at certain layers, rapidly expanding coverage
• Multi-Scale Patterns: Captures both fine-grained local context and broader document structure
• Efficiency: Achieves large effective context with minimal overhead, critical for multi-document legal reasoning

Unlike standard transformers where doubling the input length quadruples computation, Longformer's cost grows linearly. This makes processing full-length legal documents computationally feasible.

• Memory Efficiency: GPU memory usage scales linearly, preventing out-of-memory errors on long texts
• Practical Limits: Can process sequences up to 4,096 tokens (base) or beyond with custom configurations
• Benchmarking: Demonstrated on tasks like document classification and coreference resolution over entire long texts

Longformer was pre-trained from RoBERTa checkpoints with continued training on long document corpora, combining the rich linguistic knowledge of a robust encoder with the ability to handle extended context.

• Warm-Start Strategy: Initialized from RoBERTa weights, then adapted with sparse attention
• Gradual Length Increase: Sequence length increased during pre-training to stabilize learning
• Legal Fine-Tuning: The pre-trained checkpoint serves as an ideal starting point for domain-specific legal adaptation using LoRA or full fine-tuning

Beyond the encoder-only model, Longformer also offers an autoregressive variant (Longformer-Encoder-Decoder) for sequence-to-sequence tasks like long-form summarization.

• LED Architecture: Combines a Longformer encoder with a standard decoder for generation
• Global Encoder Tokens: Decoder cross-attends to global tokens in the encoder for efficient access to full source context
• Legal Summarization: Enables abstractive summarization of entire legal opinions or multi-contract analyses without truncation

The core innovation enabling linear scaling with sequence length. Unlike standard self-attention (O(n²)), Longformer uses a combination of sliding window attention for local context and global attention on pre-selected tokens (like [CLS]) to capture long-range dependencies without quadratic memory blowup. This allows processing entire contracts or briefs in a single pass.

Longformer's hybrid attention pattern combines two distinct mechanisms:

• Sliding Window: Each token attends to a fixed number of neighboring tokens (e.g., ±256), capturing local syntax and clause structure.
• Global Attention: A small set of task-specific tokens attend to all positions, acting as information highways for document-level understanding. This dual approach is critical for legal tasks requiring both clause-level precision and cross-document reasoning.

To increase the receptive field without increasing computation, Longformer employs dilated sliding windows—introducing gaps between attended tokens. This geometrically expands the context coverage while maintaining the same computational budget. For legal documents with hierarchical structure, this helps the model connect distant sections (e.g., a definition in §1.1 and its usage in §12.3) without attending to every intermediate token.

Longformer is pre-trained using a masked language modeling (MLM) objective adapted for long sequences. The model learns to predict masked tokens using both local window context and global attention signals. For legal domain adaptation, continued pre-training on corpora like PACER dockets or EDGAR filings teaches the sparse attention patterns to recognize legal-specific long-range dependencies, such as cross-referenced statutory definitions.

Longformer vs. BigBird: Both use sparse attention, but Longformer's sliding window is conceptually simpler and often faster. Longformer vs. Reformer: Reformer uses LSH-based hashing; Longformer avoids hash collisions and is more deterministic. Longformer vs. Mamba/SSMs: State-space models offer linear scaling but lack the pre-trained ecosystem of Longformer for legal NLP. For legal document processing, Longformer remains the most battle-tested long-context transformer.