Enformer Architecture

Enformer combines a convolutional tower for detecting local regulatory motifs with a transformer trunk for capturing long-range interactions. The convolutional layers first compress the 200 kb input sequence into a sequence of 1,536-channel embeddings at 128 bp resolution, reducing spatial dimensions by a factor of 128. These embeddings then feed into 7 transformer layers with 8 attention heads each, allowing the model to integrate signals from distal enhancers up to 100 kb away from target promoters.

Enformer employs skip connections that route gradients from the output heads directly to intermediate convolutional layers, bypassing the transformer trunk. This architectural choice addresses the vanishing gradient problem in deep genomic models and ensures that local motif detectors in the early convolutional layers receive strong training signals. The result is simultaneous learning of both fine-grained sequence patterns (transcription factor binding sites) and global regulatory syntax (enhancer-promoter looping logic).

The model outputs predictions for 5,313 human genomic tracks and 1,643 mouse tracks simultaneously, covering:

• CAGE-seq expression data across dozens of tissues
• DNase-seq chromatin accessibility profiles
• ChIP-seq tracks for hundreds of transcription factors
• Histone modification marks (H3K4me3, H3K27ac, etc.) Each track is predicted by a dedicated linear head operating on the final transformer embeddings, enabling shared representation learning across diverse epigenomic assays.

Standard convolutional genomic models like Basenji2 are limited to a 40 kb receptive field, missing many distal regulatory elements. Enformer's transformer layers extend the effective receptive field to 200 kb (100 kb in each direction) through self-attention, which computes pairwise interactions between all 1,536 positions in the compressed sequence. This captures enhancer-gene pairs separated by up to 100 kb, dramatically improving expression prediction for genes regulated by distal elements.

Enformer is trained jointly on aligned human and mouse genomes, leveraging synteny and evolutionary conservation. The model processes orthologous 200 kb windows from both species, with separate output heads for human and mouse tracks. This multi-species objective acts as a powerful regularizer, forcing the model to learn evolutionarily conserved regulatory grammar that generalizes better than single-species training. The shared trunk learns features that are functionally important enough to be preserved across 90 million years of divergence.

Enformer minimizes a Poisson negative log-likelihood loss appropriate for count-based sequencing data, rather than mean squared error. Each track is weighted by its sequencing coverage (total read count), ensuring that high-quality, deeply sequenced experiments contribute more to the gradient than noisy, low-coverage assays. The loss is computed as: loss = mean(coverage * (predicted - target * log(predicted))), which naturally handles the heteroscedastic noise characteristic of genomic count data.

Enformer simultaneously predicts 5,313 human and 1,643 mouse genomic tracks covering:

• CAGE-seq: Cap Analysis Gene Expression at transcription start sites
• DNase-seq: Chromatin accessibility
• ChIP-seq: Histone modifications (e.g., H3K27ac, H3K4me3)
• Transcription factor binding Each track has a dedicated linear output head operating on the final pooled representations. This multi-task setup forces the shared trunk to learn universal regulatory grammar transferable across assays and species.

Enformer's input window spans 196,608 base pairs (200 kb), a deliberate design choice to encompass the typical genomic distance between enhancers and their target promoters. The architecture combines:

• Convolutional stem: Captures local sequence motifs (e.g., TF binding sites)
• Transformer trunk: 7 self-attention layers that integrate information across the full 200 kb
• Pointwise convolution heads: Predict tracks at 128 bp resolution This receptive field is ~1.5x larger than Basenji2, directly improving enhancer-gene expression correlation predictions.

Enformer is trained jointly on human (GRCh38) and mouse (mm10) genomes, with separate output heads for each species but a shared convolutional-transformer trunk. This cross-species training acts as a powerful regularizer, leveraging evolutionary conservation of regulatory grammar. The shared trunk learns sequence features that generalize across mammals, improving prediction accuracy for both species and enabling zero-shot transfer of regulatory logic between organisms.

Enformer is trained to minimize the Poisson negative log-likelihood between predicted and observed read counts for each genomic track. This loss function is appropriate for count-based sequencing assays where the variance scales with the mean. The total loss is summed across all tracks and positions, with each track contributing equally regardless of its dynamic range. This statistical framing treats the prediction as the rate parameter of a Poisson distribution, properly modeling the discrete nature of sequencing data.