HyenaDNA

The core computational unit replacing self-attention. It combines implicit long convolutions with element-wise multiplicative gating to achieve subquadratic time complexity (O(N log N)). Unlike attention, which explicitly compares every pair of positions, the Hyena operator learns a data-controlled filter that mixes information across the sequence without materializing an N x N matrix, enabling efficient processing of megabase-length DNA.

HyenaDNA processes the genome at character-level resolution using single nucleotides (A, T, C, G) as tokens rather than fixed-length k-mers. This design choice:

• Preserves single-nucleotide polymorphism (SNP) information natively
• Avoids arbitrary k-mer boundary artifacts
• Enables fine-grained variant effect prediction directly from token probabilities
• Increases sequence length capacity by a factor of k compared to k-mer tokenization

The architecture supports a native context length of up to 1 million nucleotides, a 160x increase over standard 512-token BERT-based genomic models. This enables the model to directly learn interactions between distal regulatory elements—such as enhancers and promoters separated by hundreds of kilobases—without artificial sequence truncation or hierarchical aggregation strategies.

Instead of explicit attention weights, the Hyena operator parameterizes a long convolution kernel implicitly via a neural network. This filter is data-dependent, meaning its shape adapts to the specific input sequence. The convolution is computed efficiently in the frequency domain using the Fast Fourier Transform (FFT), reducing the computational bottleneck from O(N²) to O(N log N).

Following the long convolution, the operator applies element-wise gating across multiple heads. Each head learns a distinct gating pattern that selectively amplifies or suppresses information at each position. This mechanism serves as a form of content-based filtering, analogous to the query-key interaction in self-attention, but without computing pairwise dot products across the full sequence.

For downstream fine-tuning, HyenaDNA attaches a linear prediction head that maps the contextualized nucleotide embeddings to experimental readouts. The model predicts chromatin accessibility (DNase-seq), histone modification ChIP-seq signals, and transcription factor binding across 100+ cell types directly from raw DNA sequence, leveraging the global context to capture distal regulatory interactions.

The core computational primitive replacing self-attention in HyenaDNA. It combines implicit long convolutions with element-wise multiplicative gating to achieve subquadratic O(N log N) scaling. Unlike standard attention, which compares all token pairs, the Hyena operator learns a data-controlled filter that captures long-range dependencies without the quadratic memory cost, enabling processing of sequences up to 1 million nucleotides on a single GPU.

Rather than storing an explicit convolution kernel of length equal to the sequence, HyenaDNA parameterizes the kernel in the Fourier domain using a lightweight neural network. This network takes positional encodings as input and outputs the kernel's frequency components. The result is a sub-linear parameter count relative to sequence length, making megabase-scale genomic context computationally tractable without sacrificing the ability to model distal regulatory elements.

A critical component of the Hyena operator that provides input-dependent selectivity. After the long convolution projects the sequence, a gating network computes multiplicative masks that control information flow on a per-token basis. This gating allows HyenaDNA to filter irrelevant genomic background and amplify functionally significant motifs, mimicking the selective focus of attention without computing pairwise scores.

HyenaDNA tokenizes raw DNA sequences at the single nucleotide level (A, C, G, T) rather than using k-mer aggregation. This fine-grained representation preserves single-nucleotide polymorphism (SNP) resolution and enables precise variant effect scoring. The model learns to compose higher-order motifs from individual bases through its hierarchical layers, avoiding the vocabulary explosion and information loss associated with fixed k-mer tokenization strategies.

HyenaDNA employs a next nucleotide prediction objective during pretraining on the Human Reference Genome. This autoregressive task forces the model to learn the statistical grammar of regulatory elements, splice sites, and coding regions. The resulting representations capture evolutionary constraints and enable zero-shot variant effect prediction by comparing the log-likelihood of reference versus alternate alleles at a given locus.