Enformer Network

Enformer replaces the dilated convolutional layers used in earlier models like Basenji2 with multi-head self-attention mechanisms. This allows the model to directly weigh the influence of any nucleotide position on any other position within the 200 kb receptive field. Unlike convolutions, which aggregate information hierarchically, attention computes pairwise interactions globally, enabling the model to learn complex enhancer-promoter loops and distal regulatory syntax without a fixed inductive bias on distance.

Before the transformer layers, a convolutional neural network (CNN) stem processes the raw one-hot encoded DNA sequence. This stem consists of standard convolutional and pooling layers designed to detect local sequence motifs, such as transcription factor binding sites. This hybrid architecture leverages the CNN's strength in pattern recognition within short windows and the transformer's strength in modeling long-range dependencies, creating a hierarchical feature extraction pipeline.

Enformer outputs predictions for thousands of genomic tracks covering gene expression (CAGE-seq) and epigenomic marks (DNase-seq, ChIP-seq for histone modifications) across multiple human and mouse cell types and tissues. The model uses a shared trunk network with separate prediction heads for each track. This multi-task learning strategy forces the shared representation to capture universal regulatory grammars, improving generalization to held-out cell types and reducing overfitting on any single assay.

To manage the quadratic complexity of self-attention over very long sequences, Enformer uses a crop and aggregate strategy during training. The full 200 kb input is divided into overlapping crops, each processed independently by the transformer, and the resulting embeddings are aggregated. This technique balances the need for a large receptive field with the computational constraints of the attention mechanism, enabling training on sequences that would otherwise exceed memory limits.

Enformer is trained to predict the raw read counts from sequencing assays, which are non-negative and exhibit heteroscedasticity. The model uses a Poisson regression loss function, which is statistically appropriate for count data. This choice penalizes prediction errors relative to the expected variance, preventing the model from being dominated by high-signal regions and ensuring it learns meaningful patterns in low-coverage regulatory elements.

A key architectural validation is the model's ability to generalize across species. Enformer is trained jointly on human and mouse data, using a shared sequence-to-function mapping. The model accurately predicts regulatory activity in one species when given the orthologous sequence from the other, demonstrating that it learns evolutionarily conserved regulatory syntax. This cross-species transfer capability is a powerful test of the model's biological understanding beyond memorization.

The core paradigm that Enformer exemplifies: predicting genome-wide epigenomic tracks (chromatin accessibility, histone marks, transcription factor binding) directly from raw DNA sequence input. This approach bypasses the need for experimental assays in silico, learning the complex regulatory grammar that maps nucleotides to function. Enformer advances this by extending the receptive field to capture distal enhancer-promoter interactions up to 200 kb away, a significant leap over previous windowed models.

A training strategy where a single model simultaneously predicts multiple epigenomic assays (e.g., DNase-seq, ChIP-seq for various histone marks, CAGE-seq) across diverse cell types and tissues. Enformer is a quintessential multi-task model, outputting thousands of tracks from a shared sequence representation. This forces the network to learn a universal regulatory grammar that generalizes across assays, improving performance on data-scarce experiments through shared biological representations.

A computational perturbation technique used to interpret models like Enformer. It involves systematically introducing virtual mutations (single nucleotide variants, deletions, insertions) into a DNA sequence and measuring the predicted change in the model's output tracks. This quantifies the functional impact of non-coding variants, enabling the prioritization of candidate regulatory mutations in disease studies. Enformer's accuracy at this task allows for the fine-mapping of causal variants from GWAS loci.

The compressed, high-dimensional vector representation learned by the bottleneck layers of an autoencoder or the final embeddings of a model like Enformer. This space captures the underlying structure of complex epigenomic data, where sequences with similar regulatory functions cluster together. By analyzing the latent space, researchers can identify emergent regulatory programs, classify cell states, and generate synthetic regulatory elements, effectively decoding the 'black box' of the network.

The ability of a model trained on epigenomic data from a source set of cell types to accurately predict regulatory activity in an unseen, held-out target cell type. Enformer demonstrates strong generalization by conditioning its predictions on a learned cell type embedding. This is critical for imputing missing data in rare cell populations and suggests the model captures a fundamental regulatory code that is reused, with specific modifications, across the entire organism.