Sparse Attention

Standard self-attention scales at O(n²) with sequence length, making it prohibitively expensive for long documents. Sparse attention reduces this to O(n log n) or even O(n) by restricting the attention field. For a 100,000-token legal contract, this transforms an impossible computation into a routine one.

• Fixed Patterns: Pre-defined sparsity masks like sliding windows or dilated sliding windows
• Learned Patterns: Data-dependent sparsity where the model learns which tokens to attend to
• Memory Reduction: From gigabytes to megabytes for long-sequence processing

Pure sparse attention risks missing long-range dependencies critical in legal reasoning. Hybrid architectures combine local sparse attention for fine-grained token interactions with global attention on a few pre-selected tokens that act as information bottlenecks.

• Global Tokens: Special tokens that attend to and are attended by all other tokens, serving as a document-wide memory
• Dilated Windows: Gapped attention patterns that exponentially expand the receptive field without quadratic cost
• Task-Specific Globals: In legal summarization, global tokens can be anchored to key clauses or defined terms

Sparse attention achieves real speedups only when implemented with specialized block-sparse matrix multiplication kernels. These kernels group non-zero attention entries into contiguous blocks that map efficiently to GPU tensor cores, avoiding the overhead of scattered memory access.

• Triton Kernels: Custom GPU kernels that fuse the sparse mask application with the attention computation
• Block Size Tuning: Balancing block granularity against GPU warp utilization
• FlashAttention Integration: Combining memory-efficient exact attention with sparsity for compounded gains

The receptive field of a token in a sparse attention model is the set of all tokens whose information can reach it through the attention graph. In a legal document, a token's receptive field must encompass the entire document to capture cross-references and defined terms. Sparse patterns are designed to maximize receptive field growth.

• Logarithmic Growth: With random or dilated connections, receptive field expands exponentially with layers
• Bottleneck Analysis: Identifying tokens that act as information choke points in the attention graph
• Legal Requirement: A definition in Section 1 must be accessible to a reference in Section 50

A Transformer model employing a sparse attention mechanism that scales linearly with sequence length. It combines sliding window attention for local context with global attention on task-specific tokens like [CLS]. This hybrid pattern allows the Longformer to process entire lengthy contracts or multi-thousand-word briefs without truncation, making it a foundational architecture for legal document summarization.

A sparse-attention based Transformer designed to handle sequences up to 4096 tokens by combining three attention patterns: random attention, window attention, and global attention. This design is theoretically proven to be a universal approximator of full quadratic attention. For legal applications, BigBird excels at tasks requiring long-range dependency capture, such as identifying a clause's definition buried in a contract's preamble.

A core component of many sparse attention mechanisms where each token attends only to a fixed number of neighboring tokens on either side. This local attention pattern captures immediate syntactic and semantic context efficiently. In legal text, a sliding window of 512 tokens ensures a model understands the surrounding paragraph of a liability clause without being distracted by unrelated sections 50 pages away.

A sparse attention strategy where a small set of pre-selected tokens attend to the entire sequence, acting as information bottlenecks. Typically applied to special tokens like [CLS] or separator tokens. In multi-document legal reasoning, global attention on a case citation token allows the model to build a representation of the entire cited authority, enabling accurate cross-document alignment and fusion.

A set of metrics that automatically evaluate summary quality by counting overlapping n-grams between a candidate summary and a human-written reference. Common variants include:

• ROUGE-1: Unigram overlap
• ROUGE-2: Bigram overlap
• ROUGE-L: Longest common subsequence For legal summarization, ROUGE-L is particularly relevant as it captures sentence-level structure, though it does not measure factual consistency.

The degree to which a generated summary accurately reflects the stated facts of the source document without contradiction or fabrication. In legal contexts, this is non-negotiable. A summary stating a contract has a 30-day termination clause when the source specifies 60 days is factually inconsistent. Verification often uses Natural Language Inference (NLI) models to detect entailment or contradiction between source and summary.