Mamba

The core innovation of Mamba is its input-dependent selection algorithm. Unlike traditional SSMs that use fixed, time-invariant parameters, Mamba makes its state transition matrices (A, B, C) functions of the input at each time step. This selectivity allows the model to dynamically filter information—focusing on relevant tokens and ignoring irrelevant ones—much like an attention mechanism, but with O(N) linear complexity. The selective scan is implemented as a hardware-aware parallel scan algorithm that efficiently computes the recurrence on modern GPUs by fusing operations in SRAM, avoiding the memory-bound bottleneck of materializing large intermediate states.

Mamba's architecture is co-designed with GPU hardware constraints. The parallel scan operation is implemented using a kernel fusion strategy that performs the entire recurrence in fast SRAM rather than repeatedly reading and writing to slower HBM. Key optimizations include:

• State expansion: The latent state is expanded to a higher dimension during computation, then projected back, increasing expressivity without proportional memory cost.
• Discretization: A continuous-time SSM is discretized using a zero-order hold rule with a learnable step size Δ, which controls how much the current input influences the state.
• Work-efficient parallelism: The scan is computed in log(N) parallel steps, making training throughput competitive with highly optimized Transformer implementations.

Mamba achieves content-aware reasoning over sequences exceeding 1 million tokens without the quadratic bottleneck of self-attention. The structured state space formulation compresses the entire history into a fixed-size latent state that evolves recursively. Because the state transition is selective, the model learns to retain information from tokens deemed important and discard irrelevant context. This contrasts with:

• Transformers: O(N²) complexity, explicit key-value cache growing with sequence length.
• Traditional RNNs: Fixed state size but suffer from vanishing gradients and limited expressivity.
• Linear attention variants: Often underperform on recall-intensive tasks. Mamba's state expansion factor provides a tunable trade-off between state capacity and computational cost.

Mamba replaces the complex multi-head attention and MLP blocks of Transformers with a single, homogeneous Mamba block. Each block consists of:

• A projection layer that expands the input embedding.
• A 1D causal convolution for local feature extraction.
• A SiLU/Swish activation for non-linearity.
• The core selective SSM layer performing the recurrent computation.
• A residual connection and layer normalization. This simplification eliminates the need for positional encodings, as the recurrent nature inherently captures sequence order. The architecture scales by stacking identical blocks, with no distinction between 'encoder' and 'decoder'—it functions as a general-purpose sequence-to-sequence backbone.

Mamba demonstrates superior performance on synthetic reasoning benchmarks designed to test long-range dependency capture:

• Selective Copying: The model must memorize tokens seen earlier in a sequence and reproduce them on command, ignoring irrelevant filler. Mamba solves this with near-perfect accuracy at lengths where Transformers fail due to context window limits.
• Induction Heads: A mechanistic interpretability task where the model must learn to copy a token that follows a specific pattern. Mamba forms induction heads through its selective state updates.
• Associative Recall: Retrieving a value associated with a key seen much earlier in the sequence. The selective scan acts as a learnable key-value store compressed into the state vector. These capabilities directly translate to practical tasks like long-document QA and DNA sequence modeling.

Mamba-2 introduces the Structured State Space Duality (SSD) framework, revealing a deep mathematical connection between linear attention and state space models. Key advancements:

• Multi-head SSM: Analogous to multi-head attention, multiple smaller SSM heads operate in parallel, each with its own selective parameters, improving expressivity.
• Tensor core optimization: The SSD formulation maps the SSM computation to structured matrix multiplications that fully utilize tensor cores, achieving 2-8x faster training than Mamba-1.
• Unified theory: SSD proves that SSMs and linear attention are two views of the same underlying computation—a semiseparable matrix transformation—allowing techniques from both families to be combined.
• State expansion reparameterization: The state dimension is treated as a head dimension, enabling the model to scale to larger state capacities without architectural changes.

The dominant sequence modeling paradigm that Mamba directly challenges. Transformers use a self-attention mechanism with quadratic complexity (O(L²)) relative to sequence length L, enabling powerful long-range dependency capture but incurring prohibitive compute costs for very long sequences. Mamba's linear-time (O(L)) complexity is its primary architectural advantage, achieved by replacing explicit attention with a recurrent state space formulation that compresses context into a fixed-size hidden state.

The core mechanism Mamba's selective scan replaces. Self-attention computes a weighted sum of all input elements for every output position, creating a direct interaction path between any two tokens regardless of distance. This is both its strength (unbounded context) and weakness (quadratic cost). Mamba's selection mechanism instead learns to dynamically compress relevant history into a hidden state, making the interaction indirect but computationally tractable for sequences with millions of tokens.

The classical recurrent architecture that Mamba conceptually extends. LSTMs use gating mechanisms (input, forget, output gates) to control information flow through a cell state, mitigating the vanishing gradient problem. Mamba's structured state space formulation generalizes this gating principle: its discretized state transition matrix A and input projection B act as a continuous-time analog to LSTM gates, but with a theoretically grounded parameterization derived from the HiPPO framework for optimal memory compression.

The fundamental training pathology that state space models are designed to overcome. In deep recurrent networks, gradients shrink exponentially as they propagate backward through time, preventing the model from learning dependencies spanning more than ~100 time steps. Mamba addresses this through its HiPPO-initialized state matrix, which is mathematically constructed to memorize input history with polynomial projections, ensuring stable gradient flow across arbitrarily long sequences without the need for gating tricks or skip connections.

The process of mapping a variable-length sequence into a fixed-length vector representation. Mamba performs this implicitly: its final hidden state after processing an entire sequence serves as a learned summary embedding. Unlike attention-based pooling or mean aggregation, Mamba's embedding is generated by a selective compression process where the model learns which tokens to retain and which to discard based on their relevance to the task, making it particularly effective for sequence-level classification of long transaction histories.

The phenomenon where the statistical properties of a target variable change over time, requiring models to adapt. Mamba's selective state space mechanism provides an architectural advantage here: because its state transition parameters are input-dependent (unlike fixed-convolution models), the model can learn to reset or update its hidden state when it detects a regime change in the input stream. This makes Mamba inherently more responsive to distributional shifts in streaming transaction data than static sequence models.