FlashAttention Kernel

FlashAttention decomposes the attention matrix into smaller blocks that fit entirely within GPU SRAM. Tiling eliminates the need to materialize the full N×N attention matrix in HBM.

• Mechanism: Softmax is computed block-by-block using online rescaling, maintaining mathematical equivalence to standard attention.
• Memory reduction: Drops memory complexity from O(N²) to O(N) for a sequence of length N.
• Genomic impact: Enables processing of sequences exceeding 64k tokens on a single GPU, critical for whole-genome context windows.

Standard attention implementations launch multiple separate CUDA kernels for matrix multiply, softmax, and dropout. FlashAttention fuses these operations into a single kernel.

• Eliminates round-trips: Intermediate results stay in SRAM rather than being written to and read from HBM.
• Speedup: Achieves 2-4× wall-clock speedup over PyTorch standard attention for long sequences.
• Precision: Maintains exact attention output, not an approximation—unlike sparse or low-rank methods.

During the backward pass, FlashAttention recomputes the attention matrix and softmax statistics on-the-fly rather than storing them from the forward pass.

• Trade-off: Accepts additional FLOPs in exchange for dramatically reduced memory footprint.
• Peak memory: Reduces peak memory usage by 10-20× compared to standard attention during training.
• Genomic relevance: Makes gradient checkpointing of attention layers unnecessary, simplifying distributed training of DNA language models like HyenaDNA and Evo.

The second iteration improves GPU occupancy by redistributing work across thread blocks. FlashAttention-2 reduces the number of non-matmul FLOPs and optimizes the warp-level scheduling.

• Parallelism: Parallelizes over the sequence length dimension rather than batch and head dimensions.
• Throughput gain: Achieves up to 2× additional speedup over FlashAttention-1 on A100 GPUs.
• Utilization: Reaches up to 73% of theoretical peak FLOPs on H100 GPUs for forward pass computation.

FlashAttention-3 leverages NVIDIA Hopper architecture features including Tensor Memory Accelerator (TMA) and warpgroup-level WGMMA instructions for asynchronous data movement.

• Asynchrony: Overlaps data movement with computation using TMA to hide memory latency.
• FP8 support: Introduces low-precision FP8 attention computation for further throughput gains.
• Performance: Achieves 1.5-2.0× speedup over FlashAttention-2 on H100 GPUs, reaching 740 TFLOPS in FP16.

FlashAttention directly enables training of long-context DNA foundation models that require attention over tens of thousands of nucleotides.

• Evo model: Enables the 7B-parameter Evo model to process 131k-token genomic sequences for bacterial genome modeling.
• HyenaDNA: Complements implicit long-convolution models by accelerating the attention-based components of hybrid architectures.
• Variant effect prediction: Allows models to attend across distal regulatory elements and coding regions simultaneously, improving pathogenicity classification accuracy.

FlashAttention exploits the speed gap between High Bandwidth Memory (HBM) and Static Random-Access Memory (SRAM). HBM offers large capacity (40-80GB) but high latency, while on-chip SRAM provides ~19TB/s bandwidth with only ~20MB capacity. The kernel tiles the attention matrix into blocks that fit entirely in SRAM, computing softmax incrementally without ever materializing the full N×N matrix in HBM. This reduces HBM reads/writes from O(N²) to O(N²/M) where M is the SRAM block size.

FlashAttention employs two complementary strategies to minimize memory footprint:

• Tiling: Partitions the Q, K, V matrices into blocks that fit in SRAM. The outer loop iterates over K/V blocks, the inner loop over Q blocks, computing partial attention outputs incrementally.
• Recomputation: During the backward pass, the attention matrix and softmax statistics are not stored from the forward pass. Instead, they are recomputed on-the-fly using the stored Q, K, V and running statistics. This trades additional FLOPs for a dramatic reduction in peak memory, enabling 8-16× longer sequence lengths.

Subsequent versions introduced critical optimizations:

• FlashAttention-2: Improved work partitioning by parallelizing over the sequence length dimension rather than batch/head dimensions, reducing non-matmul FLOPs. Achieves ~2× speedup over v1 by optimizing thread block scheduling and reducing warp-level communication.
• FlashAttention-3: Targets Hopper architecture GPUs, leveraging Tensor Memory Accelerator (TMA) for asynchronous data movement and FP8 precision support. Achieves 1.5-2× throughput over v2 on H100 GPUs, critical for genomic models processing 100k+ nucleotide contexts.

FlashAttention is provably optimal in its IO complexity. The theoretical lower bound for exact attention computation requires Ω(N²d²/M) HBM accesses, where N is sequence length, d is head dimension, and M is SRAM size. FlashAttention achieves this bound asymptotically by minimizing data movement between memory hierarchy levels. This is formalized through the red-blue pebble game model of computation, establishing that no exact attention algorithm can be asymptotically more memory-efficient.

FlashAttention composes with distributed strategies for genomic-scale models:

• Tensor Parallelism: Attention heads are split across GPUs; FlashAttention reduces per-GPU memory, allowing larger head dimensions or more heads per device.
• Sequence Parallelism: Combined with Ring Attention to distribute long sequences across devices, each computing a FlashAttention block on its local sequence chunk while communicating KV blocks in a ring topology.
• ZeRO-3 Compatibility: The reduced memory footprint of FlashAttention frees capacity for larger model parameters when using ZeRO optimizer state partitioning.