Enformer

Enformer's most significant innovation is its 200 kb input window, providing a symmetric 100 kb receptive field around the central gene's transcription start site. This is a 5x increase over its predecessor Basenji2, enabling the model to directly capture distal enhancer-gene interactions that occur up to 100 kilobases away. The expanded context window allows Enformer to predict gene expression from DNA sequence alone by integrating regulatory signals from far-flanking enhancers, silencers, and insulators that previous models could not see.

Enformer employs a two-stage architecture that leverages the complementary strengths of different neural network paradigms:

• Convolutional Stem: Seven convolutional layers with progressive downsampling (factor of 128) learn local transcription factor binding motifs and sequence grammar at base-pair resolution
• Transformer Tower: Eleven transformer blocks with multi-head self-attention operate on the compressed sequence representation, modeling long-range dependencies between distal regulatory elements
• Pointwise Prediction Head: A final convolutional layer projects the transformer output to predict 5,313 genomic tracks (CAGE, ChIP-seq, DNase-seq) simultaneously at 128 bp resolution

This design separates local pattern detection from global context integration, allowing each component to specialize.

Enformer is trained on human and mouse genomes jointly, learning cross-species regulatory grammar that improves generalization. The model predicts 5,313 genomic tracks covering:

• CAGE (Cap Analysis of Gene Expression): Quantitative expression levels at transcription start sites across hundreds of human tissues and cell types
• ChIP-seq: Transcription factor binding and histone modification profiles
• DNase-seq: Chromatin accessibility maps

This multi-task setup forces the model to learn a shared internal representation of regulatory logic that generalizes across assays, tissues, and species, enabling zero-shot prediction of epigenetic marks in unseen cell types.

Enformer enables in-silico mutagenesis across the full 200 kb input window. By systematically introducing single-nucleotide variants and measuring the predicted change in gene expression, the model can:

• Score non-coding regulatory variants for functional impact
• Identify distal enhancer mutations that affect gene expression from up to 100 kb away
• Prioritize expression quantitative trait loci (eQTLs) with mechanistic hypotheses linking variant to target gene

This capability directly addresses the challenge of interpreting genome-wide association study (GWAS) hits, which predominantly fall in non-coding regions. Enformer's predictions correlate strongly with experimentally measured eQTL effect sizes.

Enformer represents a direct architectural evolution from the Basenji2 model with several critical improvements:

• Receptive field: Expanded from 40 kb to 200 kb input, increasing the effective regulatory range from ~20 kb to 100 kb
• Downsampling: Increased pooling factor from 128x to 128x but with more transformer layers (11 vs. 0) to process the compressed representation
• Attention mechanism: Introduced transformer blocks with multi-head self-attention to replace purely convolutional long-range modeling
• Training data: Trained on ENCODE and FANTOM5 consortium data covering more tissues and assay types

These changes resulted in substantially improved prediction accuracy for distal regulatory interactions while maintaining performance on local promoter prediction.

Enformer's transformer layers provide built-in interpretability through attention weight analysis. By examining the self-attention patterns, researchers can:

• Identify enhancer-gene pairs by visualizing which distal regions the model attends to when predicting expression at a target gene
• Discover novel regulatory elements that show strong attention coupling with promoters
• Validate known chromatin looping interactions (e.g., Hi-C contacts) against attention maps
• Generate contribution scores using gradient-based attribution methods (integrated gradients, DeepLIFT) to highlight specific nucleotides driving predictions

This attention-based interpretability makes Enformer a discovery tool, not just a prediction engine, enabling hypothesis generation about 3D genome organization directly from sequence.

An extension of self-attention that runs multiple attention operations in parallel, enabling Enformer to simultaneously learn different types of biological relationships within a sequence. Each attention head can specialize in distinct regulatory grammar:

• One head may focus on transcription factor binding motifs
• Another may track splice site consensus sequences
• A third may capture chromatin looping signals This parallel specialization allows the model to build rich, multifaceted representations of genomic context from raw nucleotides.

The computational task of scoring the functional impact of single-nucleotide polymorphisms (SNPs) and mutations. Enformer excels at this by predicting how a sequence change alters gene expression and epigenetic tracks across the full 200 kb context. This enables:

• Distinguishing benign variants from pathogenic ones
• Identifying non-coding drivers in cancer genomes
• Prioritizing causal variants in fine-mapped GWAS loci Enformer's predictions correlate strongly with experimental massively parallel reporter assays (MPRAs).