Epigenomic Long-Range Interactions

A convolutional technique that expands the receptive field exponentially without increasing parameter count or losing resolution. By inserting gaps between kernel elements, dilated convolutions enable a network to integrate information across tens of thousands of base pairs while preserving fine-grained sequence detail.

• Exponential receptive field growth: Each layer doubles the effective context window
• No pooling required: Maintains single-nucleotide resolution throughout the network
• Parameter efficiency: Same kernel size captures vastly larger context
• Example: A 3-layer dilated CNN with dilation rates [1, 2, 4] covers 15 base pairs; rates [1, 4, 16] cover 65 base pairs from identical kernel dimensions

The core mechanism of transformer architectures that computes pairwise interactions between all positions in a sequence simultaneously. In genomic models like Enformer, self-attention layers capture distal enhancer-promoter contacts by learning to attend across up to 200 kilobases of linear DNA.

• Global receptive field: Every position attends to every other position
• Multiple attention heads: Each head learns distinct interaction patterns (e.g., one for promoters, another for CTCF binding)
• Positional encoding: Relative position biases inform the model about genomic distance
• Quadratic complexity: O(n²) memory cost, motivating hybrid architectures that combine attention with convolutions

A computational optimization that decomposes full self-attention into separate local and global components to reduce the quadratic memory burden of long sequences. Local attention captures nearby regulatory syntax while sparse global attention connects distal elements.

• Local window attention: Restricts computation to a sliding window of 1-5 kb
• Global bottleneck tokens: A small set of learnable vectors aggregate information across the full sequence
• Strided attention patterns: Attend to every nth position for long-range skip connections
• Enables processing of megabase-scale sequences that would be intractable with dense attention

Architectural pathways that bypass one or more layers, allowing gradient flow and information to propagate directly across deep networks. In long-range genomic models, residual connections prevent the vanishing gradient problem that would otherwise prevent learning of distal regulatory relationships.

• Identity mappings: Add the input of a block directly to its output
• Pre-activation design: Normalization and activation applied before weight layers for smoother optimization
• Dense connectivity patterns: Each layer receives feature maps from all preceding layers
• Critical for training networks with 30+ layers required to capture 100 kb+ interactions

A channel-wise attention mechanism that adaptively recalibrates feature maps by modeling interdependencies between convolutional channels. In epigenomic models, SE blocks learn to emphasize biologically relevant features like transcription factor binding motifs while suppressing noise.

• Global average pooling: Compresses spatial information into channel descriptors
• Bottleneck gating: Two fully-connected layers learn channel-wise scaling factors
• Self-recalibration: The network learns which feature detectors are important for each input
• Improves sensitivity to rare regulatory elements within long input sequences

A method of injecting distance information into attention computations by biasing attention scores based on the relative separation between sequence positions. Unlike absolute encodings, relative positional biases generalize to sequence lengths unseen during training.

• Distance-aware attention: Attention weight modified by learnable bias for each relative distance
• Logarithmic binning: Distant positions grouped into coarser bins to manage parameter count
• Extrapolation capability: Model trained on 100 kb sequences can process 200 kb inputs
• Essential for capturing the distance-dependent decay of chromatin contact probability