Glossary

Enformer

A deep learning architecture that combines convolutional layers with transformer self-attention to predict gene expression and epigenetic tracks directly from 200,000 base-pair DNA sequences, explicitly modeling long-range regulatory interactions.

Get in touch Learn more

ML engineer managing model training cluster on laptop, GPU utilization visible, technical deep learning setup.

DEEP LEARNING ARCHITECTURE

What is Enformer?

Enformer is a deep neural network that predicts gene expression and epigenetic tracks directly from 200,000 base-pair DNA sequences by combining convolutional layers with transformer self-attention to explicitly model long-range regulatory interactions.

Enformer is a deep learning architecture that predicts molecular phenotypes—including gene expression and epigenetic tracks—from raw DNA sequence extending 200,000 base pairs. It integrates a convolutional stem for local motif detection with transformer self-attention blocks that capture distal enhancer-promoter interactions, dramatically improving prediction accuracy for long-range regulatory effects compared to predecessor models like Basenji2.

The architecture processes one-hot encoded DNA through seven convolutional downsampling layers that reduce spatial resolution while extracting hierarchical sequence features, then applies multi-head self-attention across the resulting 896-bp bins. This design enables Enformer to model regulatory elements up to 100 kilobases apart, making it a foundational tool for variant effect prediction and non-coding genome interpretation.

ARCHITECTURE DEEP DIVE

Key Features of Enformer

Enformer integrates convolutional layers with transformer self-attention to predict gene expression and epigenetic tracks directly from 200,000 base-pair DNA sequences, explicitly modeling long-range regulatory interactions.

Hybrid Convolutional-Transformer Architecture

Enformer combines a 7-layer convolutional tower with 11 transformer blocks to capture both local sequence motifs and distal regulatory interactions. The convolutional stem uses progressively dilated kernels to expand the receptive field efficiently before passing a compressed representation to the transformer. This hybrid design allows the model to detect transcription factor binding sites at base-pair resolution while simultaneously integrating enhancer activity from 100,000+ base pairs away.

200 kbp Input Context Window

Enformer accepts 200,000 base pairs of one-hot encoded DNA sequence as input, a dramatic expansion over earlier models like Basenji2 (131 kbp). This extended context enables the model to capture long-range enhancer-gene interactions that span tens to hundreds of kilobases. The receptive field is achieved through a combination of dilated convolutions in the stem and the global self-attention of the transformer blocks, allowing every output position to attend to the entire input sequence.

200 kbp

Input Sequence Length

5,313

Predicted Tracks (Human)

Multi-Species and Multi-Track Prediction

Enformer is trained simultaneously on human and mouse genomes, producing a shared latent representation that captures conserved regulatory grammar. The output head predicts 5,313 genomic tracks for human and 1,643 tracks for mouse, encompassing:

CAGE-seq expression data across dozens of tissues
ChIP-seq tracks for hundreds of transcription factors
DNase-seq and ATAC-seq chromatin accessibility profiles
Histone modification marks (H3K4me3, H3K27ac, etc.) This multi-task setup forces the model to learn a unified regulatory code.

Explicit Long-Range Dependency Modeling

Unlike purely convolutional models that require many layers to achieve a wide receptive field, Enformer's transformer self-attention explicitly computes pairwise interactions between all positions in the compressed sequence representation. This allows the model to directly link a distal enhancer at position 10,000 with a core promoter at position 150,000. Ablation studies show that removing the transformer layers causes a significant drop in predictive accuracy for genes regulated by far-away elements.

Basenji2 Architectural Successor

Enformer is the direct successor to Basenji2, developed by DeepMind and Calico. The key architectural advancement is the insertion of transformer blocks between the convolutional encoder and the prediction head. Basenji2 relied solely on dilated convolutions, which limited its effective receptive field. Enformer retains the multi-task regression objective and Poisson loss function from Basenji2 but achieves substantially higher accuracy on variant effect prediction tasks, particularly for expression quantitative trait loci (eQTLs) located far from their target genes.

Variant Effect Prediction via In Silico Mutagenesis

Enformer enables in silico saturation mutagenesis: by introducing a single nucleotide variant into the 200 kbp input and computing the difference in predicted tracks, researchers can estimate the regulatory impact of non-coding variants. The model's ability to capture long-range effects makes it especially valuable for prioritizing distal eQTLs and GWAS hits that fall in gene deserts. Predicted effects correlate strongly with experimentally measured allelic imbalance in reporter assays.

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

ENFORMER EXPLAINED

Frequently Asked Questions

Clear, technical answers to the most common questions about the Enformer architecture, its mechanisms, and its role in predicting gene expression from long-range DNA sequences.

Enformer is a deep learning architecture that predicts gene expression and epigenetic tracks directly from 200,000 base-pair DNA sequences by combining convolutional layers with transformer self-attention. The model processes a one-hot encoded 200kb input sequence through a series of convolutional blocks that downsample the sequence by a factor of 128, extracting local motif-level features. These features are then passed to transformer layers with multi-head self-attention, which explicitly model long-range interactions between distal regulatory elements—such as enhancers and promoters—across the full 200kb context. The architecture outputs predictions for thousands of genomic tracks, including CAGE-seq expression, DNase-seq accessibility, and ChIP-seq histone marks, at 128-bp resolution. By integrating global attention, Enformer captures regulatory syntax that purely convolutional models like Basenji2 miss, making it a foundational tool for variant effect prediction and regulatory genomics.

FOUNDATIONAL ARCHITECTURES

Related Terms

Explore the core components and related models that form the basis of the Enformer architecture, from the transformer mechanisms that capture long-range interactions to the genomic language models that inspired its design.

Transformer Self-Attention

The core mechanism Enformer uses to model interactions between distal genomic elements up to 100,000 base pairs apart. Self-attention computes a weighted representation of every position in the sequence by attending to all other positions simultaneously, allowing the model to directly capture long-range enhancer-promoter loops without the limited receptive field of convolutional layers alone.

200k bp

Input Sequence Length

O(n²)

Attention Complexity

Basenji2

The direct predecessor to Enformer, Basenji2 uses a dilated convolutional architecture to predict regulatory tracks from DNA sequence. While it achieved a receptive field of ~40kb, it lacked explicit self-attention. Enformer extended this approach by adding transformer towers after the convolutional stem, dramatically increasing the receptive field to 200kb and improving distal prediction accuracy.

EXPLORE

Multi-Head Attention

Enformer employs multiple attention heads operating in parallel, each learning a distinct relational pattern. One head might specialize in promoter-enhancer interactions, another in CTCF-mediated loop anchors, and a third in repressive chromatin domains. The outputs are concatenated and projected, giving the model a rich, multi-faceted view of regulatory syntax.

Positional Encoding

Because self-attention is permutation-invariant, Enformer must inject explicit position information. It uses relative positional encodings within the attention layers, allowing the model to reason about the distance and direction between regulatory elements. This is critical for distinguishing a promoter 50kb upstream from one 50kb downstream.

Convolutional Stem

Before the transformer towers, Enformer uses a hierarchical convolutional front-end with strided convolutions and residual blocks. This stem compresses the raw 4-channel one-hot encoded sequence into a spatially downsampled feature map, reducing the sequence length by a factor of 128 before self-attention is applied, making the quadratic attention computation tractable.

Human & Mouse Co-Training

Enformer is trained jointly on human and mouse genomes, leveraging the evolutionary conservation of regulatory grammar. The model learns to predict CAGE, DNase-seq, and ChIP-seq tracks across both species simultaneously. This multi-species objective acts as a powerful regularizer and enables cross-species transfer learning for variant effect prediction.

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.