Sparse Attention

The defining characteristic of sparse attention is the reduction of the self-attention matrix from a dense O(n²) computation to a near-linear O(n√n) or O(n log n) operation. By restricting each token to attend only to a subset of other tokens, the memory footprint drops from gigabytes to megabytes for sequences exceeding 100,000 nucleotides. This enables transformer models to process whole-genome contexts without splitting sequences into arbitrary windows that break long-range regulatory interactions.

The most intuitive sparsity pattern restricts each token's attention field to a fixed-size sliding window of neighboring tokens. For genomic sequences, a window of 1,000-10,000 nucleotides captures local regulatory grammar such as promoter-proximal elements and splice junctions. This pattern is biologically motivated: most functional elements operate within constrained genomic neighborhoods. Models like Longformer and BigBird combine local windows with global tokens to maintain long-range signal propagation.

Dilated attention introduces gaps between attended tokens, analogous to dilated convolutions, exponentially expanding the receptive field without increasing computation. A dilation rate of 2, 4, 8 across layers allows a model to capture periodic genomic features such as nucleosome positioning and chromatin periodicity. Strided attention samples tokens at regular intervals, efficiently capturing broad compositional biases like GC content and CpG island distributions across megabase-scale regions.

To prevent information isolation in purely local patterns, sparse architectures designate a small set of global tokens that attend to and are attended by all positions. In genomics, these function as learned regulatory hubs that aggregate signals from distal enhancers and propagate them to gene promoters. This mechanism mirrors the biological role of transcription hubs and enhancer clusters that coordinate gene expression through three-dimensional chromatin looping.

Beyond fixed patterns, dynamic sparse attention uses content-based routing where tokens selectively attend to the most relevant counterparts based on similarity. Techniques like Reformer's LSH attention hash tokens into buckets, while Routing Transformers use online k-means clustering. For genomic sequences, this allows the model to dynamically link homologous regions, repeat elements, and transposable element families that share sequence identity but are separated by vast genomic distances.

Block-sparse attention partitions the attention matrix into contiguous blocks and computes attention only within and between selected blocks. This pattern maps naturally to genomic data structures like topologically associating domains (TADs) and chromatin compartments, where interactions are enriched within blocks and follow specific inter-block patterns. Implementations leverage triton kernels and block-sparse matrix multiplication primitives for hardware-efficient execution on GPUs and TPUs.

The foundational mechanism that sparse attention approximates. Computes a weighted representation of every position by assessing relevance against all other positions in the sequence, resulting in O(n²) memory complexity. In genomics, this captures long-range dependencies between distal regulatory elements like enhancers and promoters, but becomes computationally prohibitive for whole-genome sequences spanning millions of base pairs.

An alternative sequence modeling architecture that processes dependencies with linear computational complexity rather than quadratic. Models like Mamba use a linear time-invariant system to selectively propagate information across extremely long sequences, making them particularly suited for whole-genome analysis where attention-based methods—even sparse ones—may still be too costly. SSMs represent a fundamentally different approach to the long-range dependency problem.

A positional encoding method that rotates query and key vectors to naturally capture relative distance between tokens. RoPE is critical for sparse attention in genomics because it enables better sequence length extrapolation—models can generalize to longer sequences than seen during training. This property pairs well with local window attention patterns, where relative positional information within the window is essential for identifying binding motifs.

A transformer-based architecture from DeepMind that predicts gene expression from DNA sequence with a 200kb receptive field. Enformer uses a combination of convolutional downsampling and transformer layers with sparse attention patterns to dramatically increase context beyond previous models. This enables capturing distal enhancer-gene interactions up to 100 kilobases away, demonstrating how sparse attention mechanisms directly enable biological discovery.

A standardized benchmark suite designed to evaluate model architectures on tasks requiring long-range sequence understanding. Includes genomics-relevant tasks like chromatin profile prediction. Sparse attention variants, SSMs, and other efficient transformers are systematically compared here, providing rigorous metrics on the accuracy-efficiency trade-off that directly informs architecture selection for genomic language models.

A visualization tool that displays the self-attention weights learned by transformer models, revealing which positions the model focuses on during prediction. In sparse attention architectures, heatmaps validate that the predefined sparsity pattern—such as local windows or dilated strides—captures biologically relevant interactions. For genomics, heatmaps can identify transcription factor binding sites and chromatin looping contacts that the model has learned to prioritize.