BigBird

BigBird replaces dense self-attention with a sparse attention pattern that scales linearly with sequence length. Instead of every token attending to every other token, each token attends to:

• Local neighbors within a sliding window
• Global tokens designated to capture document-wide context
• Random tokens to approximate full-graph connectivity This reduces memory complexity from O(n²) to O(n), making 4096-token sequences computationally tractable on standard hardware.

BigBird's attention is composed of three complementary components:

• Sliding Window Attention: Each token attends to w/2 tokens on each side, capturing local syntactic and semantic patterns
• Global Attention: A small set of tokens attend to all positions and are attended to by all positions, acting as information bottlenecks for document-level context
• Random Attention: Each token attends to a random subset of other tokens, ensuring information can propagate between distant positions This triad provably approximates full attention while maintaining linear complexity.

BigBird is grounded in graph sparsification theory and the concept of expander graphs. The model's attention pattern functions as a sparse graph where:

• The spectral gap between the first and second eigenvalues governs information flow
• The random connections ensure the graph is an expander, guaranteeing logarithmic mixing time
• This provides mathematical proof that BigBird preserves the universal approximation properties of full Transformers This theoretical rigor distinguishes BigBird from heuristic sparsification approaches.

BigBird achieves state-of-the-art results on long-document summarization benchmarks including:

• PubMed: Summarizing scientific articles with complex multi-paragraph structures
• ArXiv: Condensing technical papers while preserving key mathematical findings
• BigPatent: Generating concise descriptions of patent claims from lengthy filings In legal contexts, BigBird excels at processing full contracts, multi-page briefs, and lengthy judicial opinions without truncation, preserving critical clauses and reasoning chains that shorter-context models would miss.

BigBird adopts the standard Transformer encoder-decoder structure with its sparse attention replacing dense attention in both components:

• The encoder processes the full input document using the global-local-random pattern
• The decoder generates the summary autoregressively, attending to encoder outputs and previously generated tokens
• Global tokens in the encoder serve as aggregation points that the decoder can query for document-level understanding This architecture is particularly effective for abstractive summarization tasks where the model must generate new phrasing rather than extract existing sentences.

Both BigBird and Longformer address the long-sequence problem through sparse attention, but with key differences:

• Longformer uses a sliding window plus task-specific global attention on pre-selected tokens (e.g., CLS tokens, question tokens)
• BigBird adds random attention connections, providing stronger theoretical guarantees about information propagation
• Longformer's dilated sliding window increases receptive field without additional computation
• BigBird's random component ensures that even tokens far from global tokens can communicate efficiently For legal summarization, BigBird's random attention can be advantageous when critical information is distributed unpredictably across a lengthy contract.

A critical evaluation dimension for legal summarization. Factual consistency measures whether a generated summary accurately reflects the source document's stated facts without contradiction or fabrication.

• In legal contexts, a single hallucinated clause or misattributed ruling can have severe consequences.
• Evaluated using Natural Language Inference (NLI) models that check if summary claims are entailed by the source.
• BigBird's long context window reduces the need to split documents, minimizing the risk of losing cross-reference context that often causes factual drift.

A fallback strategy for documents exceeding even BigBird's 4096-token context window. The document is split into overlapping chunks, each summarized independently, and those summaries are then recursively condensed.

• Chunk-Level: Generate intermediate summaries for each section.
• Fusion Step: Combine intermediate summaries into a final coherent output.
• Trade-off: Risk of losing cross-chunk dependencies and global narrative structure. BigBird's extended context often eliminates the need for this lossy approach for most legal briefs and contracts.

The practice of explicitly linking each factual statement in a generated summary back to its precise location in the source document. This is non-negotiable for legal AI systems.

• Enables attorneys to rapidly verify AI outputs against original text.
• Often implemented by tracking attention weights or using post-hoc NLI verification.
• BigBird's global tokens provide natural anchor points for tracing which input segments most influenced a summary claim, supporting robust citation integrity.

Recall-Oriented Understudy for Gisting Evaluation is the standard automatic metric for summarization quality. It counts overlapping n-grams between a candidate summary and a human-written reference.

• ROUGE-1: Unigram overlap.
• ROUGE-2: Bigram overlap, capturing some fluency.
• ROUGE-L: Longest common subsequence, capturing sentence structure. While widely used, ROUGE does not measure factual consistency, so it must be paired with NLI-based faithfulness checks in legal applications.