HyenaDNA

The core innovation replacing self-attention. Instead of computing pairwise interactions between all tokens (O(L²) cost), HyenaDNA uses an implicit convolution operator parameterized by a small neural network. This operator learns a long convolution kernel that mixes information across the sequence with sub-quadratic O(L log L) complexity. The kernel is data-dependent, meaning the mixing pattern adapts to the input sequence, unlike fixed convolutional filters. This allows the model to capture long-range regulatory interactions—such as enhancer-promoter contacts spanning hundreds of kilobases—without the memory bottleneck of traditional transformers.

Unlike DNABERT or the Nucleotide Transformer which tokenize sequences into overlapping k-mers, HyenaDNA operates directly on single nucleotides as tokens. Each base (A, C, G, T) is embedded into a dense vector, giving the model a vocabulary of just 4 tokens plus special characters. This fine-grained tokenization:

• Eliminates the need to pre-select an optimal k-mer length
• Preserves single-nucleotide resolution for variant effect prediction
• Allows the model to learn its own compositional features from raw bases
• Simplifies preprocessing and avoids out-of-vocabulary tokens

The sub-quadratic complexity of implicit convolutions enables HyenaDNA to process sequences up to 1 million nucleotides in a single forward pass—roughly the length of an entire human gene locus with all its regulatory elements. This is a 2-3 order of magnitude increase over standard transformer-based genomic models, which are typically constrained to 512–4,096 tokens. The extended context allows the model to:

• Capture ultra-long-range cis-regulatory interactions
• Process entire bacterial genomes or human chromosomal segments
• Model chromatin domain-level features without arbitrary windowing
• Learn dependencies spanning megabases of linear DNA

HyenaDNA stacks multiple Hyena operator blocks, each consisting of three core components:

• Implicit long convolution: A global mixing layer that captures long-range dependencies via a learned, data-dependent kernel
• Gating mechanism: An element-wise multiplicative gate that controls information flow, similar to the gating in H3 (Hungry Hungry Hippos) architectures
• Feed-forward network: A standard position-wise MLP for local feature transformation This hierarchy interleaves global context aggregation with local non-linear processing, enabling the model to simultaneously learn short-range motifs (like transcription factor binding sites) and long-range regulatory syntax.

HyenaDNA is pre-trained on the human reference genome using a next-token prediction objective adapted for DNA. The model learns to predict the next nucleotide given the preceding context, absorbing the statistical grammar of genomic sequences—including coding regions, splice sites, repetitive elements, and regulatory motifs—without explicit annotation. The learned representations transfer effectively across species due to the conservation of fundamental genomic syntax. Fine-tuning on downstream tasks such as:

• Chromatin accessibility prediction
• Transcription factor binding site identification
• Variant effect scoring requires only a small labeled dataset and minimal compute.

By combining implicit long convolutions with gating, HyenaDNA learns a multi-scale representation of genomic sequences. The long convolution kernel captures global context—such as which chromatin compartment or topologically associating domain a region belongs to—while the gating and feed-forward layers preserve local precision for identifying individual binding motifs and splice junctions. This dual representation makes the model effective for tasks requiring both:

• Element-level prediction (e.g., variant effect at a single nucleotide)
• Sequence-level classification (e.g., promoter vs. enhancer identity) without architectural modifications between task types.

The core mathematical operator replacing self-attention in HyenaDNA. Instead of computing pairwise token interactions explicitly, it parameterizes a global convolution kernel implicitly via a neural network. This reduces the complexity from O(L²) to O(L log L), enabling the model to process sequences up to 1 million tokens in length. The kernel is data-dependent, meaning the filter adapts its shape based on the input sequence context, unlike fixed convolutional kernels.

HyenaDNA tokenizes raw DNA sequences at the character level, treating each nucleotide (A, C, G, T, N) as an individual token. This contrasts with k-mer tokenization used by DNABERT, which creates overlapping word-like units. The character-level approach:

• Eliminates vocabulary engineering and k-mer hyperparameter tuning
• Preserves single-nucleotide resolution for variant effect prediction
• Enables the model to learn its own optimal feature granularity
• Naturally handles ambiguous IUPAC codes without special preprocessing

The Hyena operator is a drop-in replacement for self-attention composed of three steps:

1. Projection: Input is projected into query, key, and value representations via linear layers
2. Implicit Convolution: The key-value interaction is modeled by a data-controlled long convolution, efficiently computed in the frequency domain using Fast Fourier Transforms
3. Gating: Element-wise multiplication with a gating signal controls information flow This decomposition preserves the input-dependent global mixing of attention while eliminating the quadratic memory bottleneck.

HyenaDNA is pre-trained on the Human Reference Genome (GRCh38) using a next-token prediction objective, but its architectural inductive biases transfer effectively across species. The model learns universal genomic grammar—motifs, periodicity, and regulatory syntax—that is conserved across eukaryotes. Fine-tuning on non-human genomes requires minimal data due to:

• Conservation of transcription factor binding site structure
• Shared chromatin organization principles
• Cross-species validity of sequence periodicity patterns like nucleosome positioning signals

Unlike convolutional neural networks with fixed kernel sizes or sparse attention with local windows, HyenaDNA's implicit long convolution provides a theoretically global receptive field across the entire input sequence. Every output position can incorporate information from all input positions. This is critical for genomics because:

• Enhancer-promoter loops can span hundreds of kilobases
• Structural variants require long-range haplotype context
• Chromatin domain boundaries are defined by distal interactions
• Co-regulated gene clusters share regulatory logic across megabases