Enformer

Enformer replaces standard convolutions with dilated convolutions arranged in an exponential tower. Each layer's dilation rate doubles, allowing the receptive field to expand to 200 kb without a proportional increase in parameters. This design captures distal enhancer-promoter interactions that shorter-range models like Basenji2 miss. The tower processes raw one-hot encoded DNA sequence (196,608 base pairs) through 7 convolutional blocks, each with progressively larger dilation gaps.

Enformer simultaneously predicts 5,313 human genomic tracks and 1,643 mouse genomic tracks in a single forward pass. These tracks span diverse data types:

• CAGE-seq: Cap Analysis Gene Expression for transcription start site activity
• ChIP-seq: Histone modification and transcription factor binding
• DNase-seq: Chromatin accessibility This multi-task setup forces shared representations to learn universal regulatory grammar, improving generalization across tissues and assays.

After the convolutional tower, Enformer applies multi-head self-attention layers to model long-range dependencies that even dilated convolutions may miss. The attention mechanism allows each genomic position to directly attend to every other position within the 200 kb window, capturing 3D chromatin looping and topologically associating domain (TAD) structures. This hybrid CNN-Transformer design combines the inductive bias of convolutions for local motif detection with the global context of attention.

Enformer's architecture is trained jointly on human and mouse genomes without explicit species labels. The model learns cross-species regulatory representations that transfer between organisms, enabling predictions of conserved regulatory elements. This design leverages evolutionary conservation as a natural regularizer: functional elements that appear in both species are reinforced during training, while species-specific noise is suppressed. The shared embedding space facilitates in silico cross-species variant effect prediction.

Enformer's final layer uses a Poisson loss function to model count-based sequencing data. Unlike mean squared error, Poisson regression properly handles the heteroscedastic noise inherent in genomic assays—where variance scales with expression level. Each track's output is a single scalar per 128-bp bin, predicting the expected read count. This statistical framing allows Enformer to naturally handle zero-inflated data common in single-cell and low-coverage epigenomic assays.

Enformer enables in silico mutagenesis by systematically mutating every nucleotide in a sequence and measuring the predicted change across all output tracks. This produces saturation mutagenesis maps that reveal regulatory motifs and predict the functional impact of non-coding variants. The model's 200 kb context window captures distal effects that shorter-range models miss, making it particularly effective for expression quantitative trait locus (eQTL) variant prioritization and rare disease diagnosis.

A precursor deep learning framework from the same research group that introduced dilated convolutional neural networks for regulatory genomics. Basenji processes 131 kb of DNA sequence to predict epigenetic tracks and gene expression. Enformer extends this architecture by expanding the receptive field to 200 kb and incorporating transformer layers for long-range attention, dramatically improving distal enhancer-gene interaction prediction.

A convolutional technique where kernel elements are spaced apart by gaps, exponentially expanding the receptive field without increasing parameter count. Enformer uses stacked dilated convolutions with rates of 1, 2, 4, 8, 16, 32, and 64 to capture regulatory interactions across 200,000 base pairs. This enables the model to link distal enhancers to their target promoters without quadratic self-attention costs over the full sequence length.

A training paradigm where a single neural network simultaneously predicts multiple related outputs from shared representations. Enformer jointly predicts 5,313 human genomic tracks (CAGE expression across tissues, DNase-seq, and ChIP-seq for histone marks) and 1,643 mouse tracks. This shared learning improves generalization, prevents overfitting, and enables the model to learn universal regulatory grammar transferable across species.

A computational perturbation method where every nucleotide in an input sequence is systematically mutated to measure the predicted change in model output. With Enformer, this reveals causal regulatory variants by quantifying how single nucleotide changes alter predicted gene expression. The technique identifies transcription factor binding motifs and enhancer logic without requiring wet-lab experiments, accelerating variant interpretation for clinical genomics.

A mechanism that injects information about token order into input embeddings. Enformer uses relative positional encodings within its transformer layers to capture the spatial relationships between regulatory elements. Unlike absolute encodings, relative encodings allow the model to generalize to sequence lengths not seen during training and better represent the distance-dependent decay characteristic of enhancer-promoter interactions.

Genomic loci where genetic variants are statistically associated with changes in mRNA expression levels. Enformer's predictions correlate strongly with GTEx eQTL data, demonstrating that the model captures causal regulatory variants from sequence alone. By computing the predicted expression difference between reference and alternate alleles, Enformer provides in silico variant effect scores that prioritize functional non-coding variants for experimental validation.