Gradient checkpointing is a memory-for-compute trade-off technique used during the backward pass of neural network training. Instead of storing all intermediate activations from the forward pass—which consumes memory proportional to network depth—it strategically saves only a subset (checkpoints). The non-saved activations are recomputed on-demand from the nearest checkpoint during backpropagation, dramatically reducing peak memory usage at the cost of extra computation.
Reference
Gradient Checkpointing
Gradient checkpointing is a memory optimization technique for training deep neural networks that trades compute for memory by selectively recomputing intermediate activations during the backward pass instead of storing them all.
MEMORY COMPRESSION TECHNIQUE
What is Gradient Checkpointing?
Gradient checkpointing is a memory optimization technique for training deep neural networks that trades compute for memory by selectively recomputing intermediate activations during the backward pass instead of storing them all.
This technique is critical for training extremely large models or processing long sequences where memory is the primary constraint. It is often implemented via automatic differentiation frameworks like PyTorch's torch.utils.checkpoint. Related memory optimization paradigms include model parallelism and the Zero Redundancy Optimizer (ZeRO), which address memory redundancy across distributed devices.
MEMORY COMPRESSION TECHNIQUE
Key Characteristics of Gradient Checkpointing
Gradient checkpointing is a memory-for-compute trade-off technique that strategically recomputes intermediate neural network activations during backpropagation to drastically reduce peak memory consumption during training.
01
Core Trade-Off: Compute for Memory
Gradient checkpointing fundamentally trades increased computation for reduced memory. Instead of storing all intermediate activations from the forward pass for the backward pass, it selectively stores only a subset (checkpoints). The non-stored activations are recomputed on-demand during backpropagation from the nearest checkpoint. This reduces peak memory usage from O(n) to O(√n) for a chain of n layers, where n is the sequence length or depth.
02
Strategic Checkpoint Placement
GRADIENT CHECKPOINTING
Frequently Asked Questions
Gradient checkpointing is a critical memory optimization technique for training large neural networks. This FAQ addresses its core mechanisms, trade-offs, and practical applications in modern AI engineering.
Gradient checkpointing is a memory optimization technique for training deep neural networks that trades compute for memory by selectively recomputing intermediate activations during the backward pass instead of storing them all. During the standard forward pass, the network computes and typically stores every layer's output (activation) to use later for calculating gradients. With checkpointing, only a strategically chosen subset of these activations (the checkpoints) are stored. During the backward pass, when gradients for a non-checkpointed layer are needed, the network re-executes the forward pass for that segment of the model, starting from the nearest stored checkpoint, to regenerate the required activations on-the-fly. This process reduces peak memory consumption from O(n) to O(√n) (where n is the number of layers) at the cost of approximately one additional forward pass per backward pass.