Glossary

Enformer

A transformer-based deep learning architecture that predicts gene expression and epigenetic tracks from 200-kilobase DNA sequences, explicitly modeling long-range enhancer-promoter interactions through multi-head attention mechanisms.

Get in touch Learn more

ML engineer managing model training cluster on laptop, GPU utilization visible, technical deep learning setup.

GENE EXPRESSION PREDICTION

What is Enformer?

Enformer is a transformer-based deep learning architecture that predicts gene expression and epigenetic tracks directly from 200-kilobase DNA sequences, explicitly modeling long-range enhancer-promoter interactions through multi-head attention mechanisms.

Enformer is a deep neural network that processes raw DNA sequences up to 200 kilobases in length to simultaneously predict thousands of molecular phenotypes, including RNA expression levels and chromatin accessibility tracks. Its core innovation is replacing dilated convolutions with multi-head self-attention layers, enabling the model to directly learn interactions between regulatory elements separated by vast genomic distances without the inductive bias of local receptive fields.

The architecture first tokenizes one-hot encoded DNA into patches, processes them through convolutional stem layers, then applies a stack of transformer blocks with relative positional encodings. This design captures enhancer-promoter looping and other distal cis-regulatory mechanisms that previous convolutional models like Basenji2 could not fully resolve, achieving state-of-the-art performance on predicting expression quantitative trait loci and variant effects from the reference genome alone.

ARCHITECTURE DEEP DIVE

Key Architectural Features

The Enformer architecture departs from standard convolutional stacks by integrating transformer modules to explicitly model long-range dependencies across 200-kilobase genomic sequences.

Multi-Head Attention for Distal Interactions

The core innovation of Enformer is the replacement of dilated convolutions with multi-head self-attention layers in the model's trunk. This mechanism computes pairwise interaction scores between every position in the 200kb input sequence, allowing the model to directly learn enhancer-promoter communication without being constrained by a limited receptive field. Each attention head can specialize in different regulatory grammar rules, such as cell-type-specific looping or insulator bypassing.

200 kb

Input Sequence Length

5,313

Predicted Tracks (Human)

Convolutional Stem and Tokenization

Before the transformer trunk, a convolutional stem compresses the raw one-hot encoded DNA sequence. This stem uses strided convolutions and pooling to reduce the 200,000-base-pair sequence into a manageable sequence of 1,536-dimensional token embeddings. This step is critical for computational feasibility, as the quadratic complexity of self-attention would be prohibitive on raw nucleotide resolution. The stem learns local motif detectors analogous to those in Basenji2.

Multi-Scale Species Prediction Heads

Enformer employs a multi-task learning framework with separate prediction heads for human and mouse genomes. The output layer predicts thousands of epigenetic tracks (CAGE, ChIP-seq, DNase-seq) across diverse cell types. A key architectural detail is the use of cropping and assembly to predict on sequences longer than the 200kb input window, enabling genome-wide inference by stitching overlapping predictions together.

Relative Positional Encodings

To capture the precise spatial grammar of regulatory elements, Enformer utilizes relative positional encodings within its attention layers. Unlike absolute positional encodings, this method injects distance-aware biases into the attention logits, allowing the model to learn that a binding site 50 kilobases away has a different regulatory logic than one 5 kilobases away. This is essential for modeling the directional specificity of enhancer activity.

Stochastic Depth and Regularization

To train such a deep and wide architecture on high-dimensional genomic data, Enformer integrates stochastic depth dropout (dropping entire transformer layers during training) and pre-activation residual connections. These techniques prevent co-adaptation of features and ensure smooth gradient flow through the 7 transformer layers stacked between the convolutional stem and the prediction heads, enabling stable convergence on the massive training dataset.

ARCHITECTURE COMPARISON

Enformer vs. Basenji2 vs. DeepSEA

Comparison of three deep learning architectures for predicting gene expression and epigenomic profiles from DNA sequence.

Feature	Enformer	Basenji2	DeepSEA
Input Sequence Length	200 kb	131 kb	1 kb
Core Architecture	Transformer + CNN	Dilated CNN	CNN
Attention Mechanism
Long-Range Enhancer-Promoter Modeling
Output Resolution	128 bp bins	128 bp bins	200 bp bins
Number of Predicted Tracks	5,313	4,229	919
Multi-Task Learning
Training Dataset	ENCODE + FANTOM5	ENCODE + Roadmap	ENCODE + Roadmap

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

ENFORMER EXPLAINED

Frequently Asked Questions

Clear, technical answers to the most common questions about the Enformer architecture, its mechanisms, and its role in predicting gene expression from long-range DNA sequences.

Enformer is a transformer-based deep learning architecture that predicts gene expression and epigenetic tracks directly from 200-kilobase DNA sequences. It works by first processing the one-hot encoded input sequence through a series of convolutional layers with progressively increasing dilation rates to capture local motif patterns, then feeding the resulting embeddings into multi-head self-attention layers that explicitly model long-range interactions between distal regulatory elements, such as enhancers and promoters. The architecture outputs predictions for thousands of genomic tracks—including CAGE-seq expression and ChIP-seq histone marks—across multiple human and mouse cell types and tissues. By increasing the receptive field from 131 kb (as in its predecessor Basenji2) to 200 kb, Enformer captures regulatory interactions that span vast genomic distances, dramatically improving the accuracy of variant effect predictions on gene expression.

ARCHITECTURE & CONTEXT

Related Terms

Core concepts for understanding the Enformer architecture and its role in predicting gene expression from long-range DNA interactions.

Basenji

The direct predecessor to Enformer, Basenji uses dilated convolutional neural networks to predict epigenetic and transcriptional profiles from 131-kilobase DNA sequences. It captures long-range regulatory interactions without explicit chromatin contact data, laying the groundwork for the receptive field expansion achieved by Enformer's transformer-based design.

EXPLORE

Multi-Head Attention

The core mechanism enabling Enformer to model long-range enhancer-promoter interactions across 200-kilobase sequences. Unlike convolutional layers with fixed receptive fields, multi-head attention computes pairwise interactions between all positions in the sequence simultaneously, allowing the model to learn which distal regulatory elements influence a gene's expression regardless of genomic distance.

Cis-Regulatory Elements

Non-coding DNA regions—including enhancers, promoters, and silencers—that regulate transcription of genes on the same chromosome. Enformer explicitly models how these elements interact over long genomic distances. Key types include:

Enhancers: Activate transcription from distal locations
Promoters: Initiate transcription at gene start sites
Silencers: Repress transcription in specific contexts

Multi-Task Learning

Enformer employs a multi-task training paradigm where a single neural network simultaneously predicts thousands of epigenetic and transcriptional tracks across multiple human and mouse cell types. By sharing hidden representations, the model leverages common regulatory logic across assays, improving generalization and enabling predictions for experimentally uncharacterized cell types.

In Silico Mutagenesis

A computational perturbation method used to interpret Enformer's predictions. By systematically introducing virtual nucleotide substitutions into a 200-kilobase input sequence and measuring the resulting change in predicted gene expression, researchers can identify causal regulatory variants and quantify the functional impact of non-coding mutations associated with disease.

One-Hot Encoding

The numerical representation scheme used to convert raw DNA sequences into Enformer's input format. Each nucleotide (A, C, G, T) is mapped to a binary vector of length four with a single high bit, transforming a 200-kilobase character string into a sparse matrix of shape (200,000 × 4). This encoding preserves the discrete nature of nucleotides without imposing artificial ordinal relationships.

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.