Long-Range Dependencies

The quintessential example of a long-range dependency. Enhancers are regulatory DNA elements that can activate a gene's promoter from distances exceeding 1 megabase. DNA looping brings these distal elements into physical proximity, bypassing intervening sequence. Models must learn that a binding event at an enhancer controls transcription initiation at a distant, non-adjacent promoter.

Standard self-attention computes pairwise interactions between all positions, scaling quadratically with sequence length (O(L²)). For a 100kb genomic locus, this requires 10 billion comparisons per layer. This computational cost is the primary barrier to modeling whole-genome context. Solutions include:

• Sparse attention patterns that restrict the field of view
• Linear attention approximations that avoid the full pairwise matrix
• State space models like Mamba that scale O(L)

Genes involved in the same biological pathway are often clustered in syntenic blocks—conserved chromosomal neighborhoods across species. These clusters are maintained by long-range regulatory interactions that coordinate expression. A genomic model must recognize that disrupting a boundary element can cause an enhancer to inappropriately activate a neighboring gene, a mechanism underlying some cancers and developmental disorders.

TADs are megabase-scale chromatin structures where DNA interactions occur preferentially within the domain boundary. They act as regulatory neighborhoods that constrain enhancer-promoter communication. Key characteristics:

• Boundaries are enriched for CTCF binding sites
• Disruption of TAD boundaries can cause enhancer adoption and gene misexpression
• Models must learn that linear distance is a poor proxy for functional distance within a TAD

Transformers lack inherent sequence order awareness, requiring positional encodings to inject location information. Standard sinusoidal encodings struggle with very long sequences. Advanced methods like Rotary Position Embedding (RoPE) encode relative position through rotation matrices, enabling better extrapolation to unseen lengths. For genomic models, this means a variant 500kb away can still be precisely located relative to a gene.

An alternative to attention for capturing long-range dependencies. Dilated convolutions expand the receptive field exponentially with layer depth by inserting gaps between kernel elements. The Hyena operator combines long convolutions with element-wise gating to achieve subquadratic scaling while maintaining global context. This allows processing of megabase-length sequences without the memory cost of full attention.

The core Transformer component that computes a weighted representation of every position in a sequence by comparing all pairs of positions. In genomics, this allows a model to directly relate a distal enhancer at position 1,000,000 with a promoter at position 500,000 without sequential processing through intermediate nucleotides. The attention matrix explicitly models pairwise interactions, making it the primary vehicle for capturing long-range dependencies, though its O(L²) memory complexity becomes prohibitive for whole-genome contexts.

A position encoding technique that encodes absolute position information into the attention computation via rotation matrices. Unlike absolute or learned positional embeddings, RoPE naturally captures relative distance between tokens through the dot-product operation in attention. This property is critical for genomic models because it enables them to extrapolate to sequence lengths unseen during training, allowing a model trained on 100kb sequences to generalize to megabase-scale inputs where long-range regulatory interactions occur.

An input-output exact attention algorithm that minimizes high-bandwidth memory (HBM) reads and writes by fusing operations in a single CUDA kernel. For genomic models processing extremely long DNA sequences, FlashAttention reduces the memory footprint from quadratic to linear in sequence length, enabling training on contexts up to 64k tokens or more on a single GPU. This hardware-aware optimization makes capturing long-range dependencies between distal genomic elements computationally tractable without sacrificing exact attention.

A subquadratic sequence mixing operator that replaces attention with a combination of long convolutions and element-wise gating. Unlike standard self-attention, the Hyena operator scales in O(L log L) time, enabling genomic models to process sequences up to 1 million nucleotides in length. This efficiency allows the model to implicitly learn long-range dependencies across entire gene loci, including interactions between promoters, enhancers, and silencers, without the quadratic bottleneck of pairwise attention.

A structured state space sequence model with a selection mechanism that allows the model to filter information based on the input content. Unlike static state space models, Mamba's input-dependent parameters enable it to selectively propagate or forget information across long sequences, offering linear-time scaling as an alternative to attention. For genomic sequence analysis, this means the model can efficiently track regulatory signals across hundreds of kilobases while maintaining a compact hidden state.