FlashAttention

The core innovation of FlashAttention is decomposing the attention computation into small tiles that fit entirely into on-chip SRAM. By fusing the softmax reduction across these tiles using a numerically stable online algorithm, the method avoids materializing the full N×N attention matrix in HBM. This reduces the number of high-latency HBM reads and writes from quadratic O(N²) to sub-quadratic, directly addressing the memory bottleneck that plagues standard attention implementations on long sequences.

Unlike sparse or low-rank attention approximations that trade accuracy for speed, FlashAttention computes mathematically exact attention scores. The tiling and online softmax algorithm is numerically equivalent to the standard implementation, producing identical outputs up to floating-point precision. This guarantees no degradation in model quality or perplexity, making it a drop-in replacement that accelerates training and inference without requiring architectural changes or retuning hyperparameters.

Standard attention stores the intermediate N×N attention matrix for the backward pass, causing memory consumption to scale quadratically with sequence length. FlashAttention employs recomputation in the backward pass: it recalculates attention scores on-the-fly from the stored softmax normalization statistics rather than reading the full matrix from HBM. This reduces peak memory usage from O(N²) to O(N), enabling training with sequence lengths up to 8K or 16K tokens on hardware that would otherwise run out of memory.

FlashAttention is implemented as a single, highly optimized CUDA kernel that fuses multiple operations—matrix multiply, softmax, dropout, and masking—into one GPU pass. This eliminates the overhead of launching separate kernels and storing intermediate results in global memory. The fused kernel maximizes occupancy and arithmetic intensity by keeping data in registers and shared memory, achieving up to 7.6x speedup over standard PyTorch attention and approaching the theoretical peak throughput of modern NVIDIA GPUs.

The second iteration, FlashAttention-2, introduces critical work partitioning improvements. It parallelizes over the sequence length dimension rather than the batch and head dimensions, achieving higher GPU occupancy. It also reduces non-matmul FLOPs by optimizing the rescaling logic within the online softmax loop. These changes bring utilization closer to theoretical peak and deliver approximately 2x speedup over FlashAttention-1 on modern architectures like the H100 GPU, particularly for the forward pass.

FlashAttention-3 is purpose-built for the NVIDIA Hopper GPU architecture (H100/H200). It leverages new hardware features such as WGMMA (Warp Group Matrix Multiply-Accumulate) instructions for asynchronous tensor core operations and TMA (Tensor Memory Accelerator) for hardware-accelerated data copying between global and shared memory. By overlapping computation with data movement using Hopper's asynchronous execution model, FlashAttention-3 achieves up to 1.5-2.0x speedup over FlashAttention-2 and reaches 740 TFLOPS on H100 GPUs, representing 75% of theoretical peak FP16 performance.