Glossary

Hyena Operator

A subquadratic sequence mixing operator that replaces attention with a combination of long convolutions and element-wise gating, enabling genomic models to process megabase-length sequences efficiently.

Get in touch Learn more

Stylish WeWork-like workspace with hot desks and document wall, professional searching through enterprise knowledge base on a mounted ultrawide display, warm industrial pendants overhead.

SUBQUADRATIC SEQUENCE MIXING

What is Hyena Operator?

The Hyena operator is a sequence mixing mechanism that replaces standard self-attention with a combination of implicit long convolutions and element-wise gating, achieving subquadratic computational complexity.

The Hyena operator is a subquadratic sequence mixing primitive that replaces the quadratic-cost self-attention mechanism in Transformer architectures. It interleaves implicit long convolutions—parameterized efficiently via fast Fourier transforms—with element-wise multiplicative gating, enabling the model to capture long-range dependencies while scaling in near-linear time relative to sequence length.

By avoiding the explicit computation of an N×N attention matrix, the operator drastically reduces memory footprint and compute cost for long sequences. In genomics, this allows models like HyenaDNA to process up to 1 million nucleotides in a single context window, learning regulatory interactions between distal genomic elements such as enhancers and promoters without arbitrary truncation.

Subquadratic Sequence Modeling

Key Features of the Hyena Operator

The Hyena operator replaces quadratic self-attention with a subquadratic mix of long convolutions and element-wise gating, enabling genomic models to process megabase-length sequences efficiently.

Subquadratic Complexity

The Hyena operator achieves O(N log N) time complexity by replacing pairwise attention with implicit convolutions. This eliminates the O(N²) bottleneck of standard self-attention, making it computationally feasible to process entire megabase-length genomic sequences in a single forward pass without truncation or chunking.

O(N log N)

Time Complexity

1M+

Max Sequence Length (nucleotides)

Hybrid Convolution-Gating Architecture

Hyena decomposes the attention operator into two core primitives:

Long Convolutions: Implicitly parameterized convolutions that efficiently mix information across vast sequence distances, capturing long-range dependencies like enhancer-promoter interactions.
Element-wise Gating: Input-dependent multiplicative gates that modulate information flow, providing the content-awareness typically supplied by attention's query-key dot products.

Implicit Convolution Parameterization

Rather than learning a massive explicit convolution kernel, Hyena parameterizes its long convolutions implicitly through a smaller neural network. This network maps positional indices to kernel values, dramatically reducing parameter count while maintaining the ability to model global sequence interactions across hundreds of thousands of tokens.

HyenaDNA: Genomic-Scale Application

HyenaDNA is a genomic foundation model built natively on the Hyena operator. It processes sequences up to 1 million nucleotides in length—encompassing entire bacterial genomes or large human genomic loci—enabling whole-context learning without the fragmentation required by Transformer-based models. This unlocks zero-shot variant effect prediction across entire genes in a single pass.

EXPLORE

Recursive Operator Decomposition

The Hyena operator is constructed by recursively composing fast convolution and gating layers. Each order of the recursion increases the expressivity of the mixing operation. A second-order Hyena operator approximates the behavior of self-attention while maintaining subquadratic scaling, striking a balance between model fidelity and computational efficiency.

Global Context Without Positional Bottlenecks

Unlike linear attention variants that compress the past into a fixed-size state, Hyena's long convolutions maintain a theoretically unbounded receptive field without summary bottlenecks. This is critical for genomics, where regulatory elements can act over distances exceeding 100 kilobases, requiring models to retain fine-grained information across vast genomic spans.

SEQUENCE MIXING ARCHITECTURES

Hyena Operator vs. Attention vs. State Space Models

Comparative analysis of core computational mechanisms for processing long genomic sequences, evaluating complexity, memory footprint, and biological context capture.

Feature	Hyena Operator	Self-Attention	State Space Models (Mamba)
Computational Complexity	Subquadratic O(N log N)	Quadratic O(N²)	Linear O(N)
Core Mechanism	Long convolution + element-wise gating	Pairwise token comparison via QKV	Linear time-invariant system with selection
Maximum Context Length (Genomic)	1,000,000+ nucleotides	8,000-128,000 nucleotides	1,000,000+ nucleotides
Memory Footprint	Low	High	Very Low
Implicit Convolution
Content-Aware Filtering
Pairwise Interaction Modeling	Implicit via convolution	Explicit via attention matrix	Implicit via hidden state
Training Throughput (Long Sequences)	High	Low	Very High

HYENA OPERATOR

Frequently Asked Questions

Clear, technically precise answers to the most common questions about the Hyena operator, its subquadratic scaling, and its transformative role in long-range genomic sequence modeling.

The Hyena operator is a subquadratic sequence mixing operator designed as a drop-in replacement for the standard self-attention mechanism in Transformer architectures. It works by combining implicitly parametrized long convolutions with element-wise multiplicative gating. Instead of computing pairwise interactions between all tokens—which scales quadratically with sequence length—Hyena learns a data-controlled linear operator. Specifically, it generates a convolution filter dynamically from the input sequence, applies it efficiently via the Fast Fourier Transform, and then modulates the result with a gating mechanism. This reduces the computational complexity from O(L²) to O(L log L), enabling the processing of sequences with hundreds of thousands to millions of tokens, such as entire human chromosomes, without the memory bottleneck of attention matrices.

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

HYENA OPERATOR ECOSYSTEM

Related Terms

Explore the core components, alternative architectures, and foundational models that define the subquadratic sequence modeling landscape for genomics.

Long Convolutions

The primary sequence mixing mechanism within the Hyena operator, replacing the quadratic pairwise comparisons of self-attention. Unlike standard convolutions with small kernels, long convolutions use kernels as long as the input sequence itself, efficiently parameterized in the frequency domain via the Fast Fourier Transform (FFT). This allows the model to capture long-range dependencies across megabase-scale DNA sequences without explicitly computing an attention matrix.

O(N log N)

Computational Complexity

Element-wise Gating

The second critical component of the Hyena operator, applied after the long convolution. Gating involves multiplying the convolution output by a context-dependent, non-linear gate. This mechanism acts as a dynamic filter, allowing the model to selectively amplify or suppress specific signals based on the input content. In genomics, this enables the model to condition its processing on the presence of specific transcription factor binding motifs.

HyenaDNA

A genomic foundation model that implements the Hyena operator to process DNA sequences. By replacing quadratic attention with subquadratic convolutions and gating, HyenaDNA extends the effective context window to 1 million nucleotides at single-nucleotide resolution. This allows the model to directly learn interactions between distal regulatory elements, such as enhancers and promoters, across an entire gene locus without truncation or binning.

EXPLORE

Mamba State Space Model

A competing subquadratic architecture that serves as an alternative to the Hyena operator. Mamba is a structured state space model (SSM) with a selection mechanism that makes its parameters input-dependent. While Hyena uses explicit long convolutions, Mamba models sequences through a continuous-time state space representation, achieving linear time complexity O(N). Both architectures are key contenders for replacing attention in ultra-long genomic sequence analysis.

Self-Attention Mechanism

The incumbent sequence mixing technology that the Hyena operator is designed to replace for long-range tasks. Self-attention computes a weighted representation of every position by comparing all pairs of positions, resulting in quadratic O(N²) time and memory complexity. For a 1-megabase DNA sequence, this creates a matrix with 10¹² entries, making it computationally prohibitive. Hyena's subquadratic scaling directly addresses this bottleneck.

O(N²)

Attention Complexity

Implicit Convolutions

A parameterization technique used within the Hyena operator to generate the long convolution kernel. Instead of learning the kernel weights directly, an implicit convolution uses a small neural network, often an MLP, to produce the kernel values as a function of the input. This decouples the number of parameters from the sequence length, enabling the model to scale to megabase contexts without a proportional increase in memory footprint.

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.