FlashAttention

FlashAttention's core innovation is IO-awareness—it accounts for the speed hierarchy between GPU memory levels. Instead of materializing the full N×N attention matrix in slow High Bandwidth Memory (HBM), it:

• Tiles the Q, K, and V matrices into blocks that fit in fast SRAM
• Computes softmax attention incrementally within each block
• Writes only the final output back to HBM

This reduces HBM reads/writes from O(N²) to O(N²/d), where d is the head dimension. For a 4K-token legal document, this can mean a 7.6x reduction in memory operations.

Traditional softmax requires two passes over the data: one to find the maximum value for numerical stability, and another to compute the exponential sum. FlashAttention fuses these into a single online softmax algorithm that:

• Maintains running statistics (m, ℓ) as it iterates through blocks
• Rescales previously computed partial sums when a new maximum is encountered
• Produces mathematically identical output to the two-pass version

This is critical for legal models, where attention scores across long documents can have extreme dynamic ranges, and numerical precision directly impacts citation accuracy.

In the backward pass, standard attention implementations store the full attention matrix from the forward pass, consuming O(N²) memory. FlashAttention recomputes attention scores on-the-fly during backpropagation:

• Stores only the softmax normalization statistics (m, ℓ) from the forward pass
• Recalculates QK^T values in SRAM during the backward pass
• Trades a small increase in FLOPs for a massive reduction in memory footprint

This recomputation strategy enables training on 4-8x longer sequences than standard attention, directly enabling processing of full contracts and multi-page judicial opinions without truncation.

FlashAttention-2 improves upon the original by optimizing work partitioning across thread blocks and reducing non-matmul FLOPs:

• Forward pass: Parallelizes over the sequence length dimension instead of batch and head dimensions, increasing occupancy
• Backward pass: Parallelizes over the column dimension, eliminating the need for atomic adds and inter-thread communication
• Reduces the number of non-matrix-multiply operations, which are disproportionately slow on modern GPUs

These optimizations yield a 2x speedup over FlashAttention-1, making long-document legal pre-training economically viable on existing GPU clusters.

FlashAttention-3 targets NVIDIA's Hopper GPU architecture (H100) with three new techniques:

• Warp-specialization: Dedicated warps for data movement (TMA) and computation, overlapping asynchronous loads with math
• Ping-pong scheduling: Alternates between two sets of registers to hide memory latency
• FP8 low-precision: Leverages Hopper's native FP8 tensor cores for attention computation, further reducing memory bandwidth

On H100 GPUs, FlashAttention-3 achieves 1.5-2.0x speedup over FlashAttention-2, reaching up to 740 TFLOPS—critical for training legal models on entire case law corpora within reasonable timeframes.

The memory footprint of standard attention scales quadratically with sequence length—a 16K-token legal document requires 256x more attention memory than a 1K-token passage. FlashAttention's memory scales linearly with sequence length:

• Standard Attention: O(N²) memory for the attention matrix
• FlashAttention: O(N) memory for the output and statistics
• Enables processing of 64K+ token sequences on a single GPU

This linear scaling is the key enabler for domain-adaptive pre-training on full legal documents. Without it, models must truncate or chunk documents, breaking cross-reference reasoning between distant sections of a contract or opinion.

The foundational design principle behind FlashAttention. Traditional attention is compute-bound on short sequences but memory-bound on long ones. An IO-aware algorithm minimizes high-bandwidth memory (HBM) reads and writes by fusing operations in fast SRAM. Key mechanism: Tiling the attention matrix into blocks that fit entirely in SRAM, computing softmax incrementally without materializing the full N×N matrix. This reduces HBM access from O(N²) to O(N²/M) where M is SRAM size.

A numerically stable algorithm for computing softmax over a sequence incrementally, without storing the entire input vector. FlashAttention uses a variant called tiled online softmax that processes attention scores in blocks. How it works: Maintains running statistics (max value and sum) across tiles, rescaling previous partial results when a new maximum is discovered. This is what enables exact attention computation without the O(N²) memory footprint of storing the full attention matrix.

Understanding FlashAttention requires knowing the GPU memory stack. HBM (High Bandwidth Memory): Large capacity (40-80GB on A100/H100) but high latency. SRAM (Static RAM): Small (192KB per streaming multiprocessor on A100) but ~10x faster bandwidth. Standard attention writes the full attention matrix to HBM. FlashAttention keeps computation in SRAM by splitting the Q, K, V matrices into tiles, computing partial softmax results, and only writing the final output to HBM.

A critical design choice in FlashAttention that trades compute for memory. During the forward pass, the algorithm does not store the full attention matrix for the backward pass. Instead, it stores only the softmax normalization statistics (m and l). During backpropagation, the attention scores are recomputed on-the-fly from the stored Q, K, V tiles. This avoids the O(N²) memory cost of caching attention weights while remaining numerically exact. The recomputation overhead is offset by the massive memory savings.

Successive iterations improved on the original algorithm. FlashAttention-2 (2023) reduced non-matmul FLOPs by restructuring the loop order to parallelize over the sequence length dimension, achieving ~2x speedup on A100 GPUs. FlashAttention-3 (2024) leverages Hopper architecture features: asynchronous execution of TMA (Tensor Memory Accelerator) and WGMMA (Warp Group Matrix Multiply-Accumulate) instructions, plus FP8 low-precision computation, reaching 1.5-2.0x over FlashAttention-2 on H100 GPUs.

An extension combining IO-awareness with sparsity for further speedups. Instead of computing all pairwise attention scores, a sparsity mask identifies which blocks of the attention matrix are likely to contain meaningful connections. Common patterns include sliding window (local attention), global tokens, and dilated sliding window. Block-sparse FlashAttention skips entire tile computations, reducing the effective sequence length. This is particularly useful for legal documents where attention can be localized to sections.