Gradient Checkpointing

During a standard forward pass, all intermediate activations are stored in memory for the backward pass. Gradient checkpointing breaks this convention by discarding activations of designated layers. During backpropagation, when a discarded activation is needed for the gradient calculation, it is recomputed on-demand by re-running a segment of the forward pass. This transforms the memory footprint from O(n) to O(sqrt(n)) or O(log n) for the checkpointed segments, at the cost of approximately 33% additional forward computation.

In Vision Transformer (ViT) architectures, checkpointing is typically applied to the MLP and self-attention blocks within each encoder layer. Frameworks like PyTorch implement this via torch.utils.checkpoint.checkpoint, which wraps a module segment as a no-gradient function. During the backward pass, the autograd engine triggers a re-computation of the wrapped segment with gradients enabled. This is particularly critical for Swin Transformers and Masked Autoencoders (MAE) processing large 3D medical volumes where activation memory can exceed 40GB per sample.

A standard n-layer network stores M activations. With optimal checkpointing at every sqrt(n) layers, the memory requirement drops to approximately O(sqrt(n) * M). For a ViT-Large model with 24 transformer blocks processing a 3D CT scan:

• Without checkpointing: ~36GB activation memory
• With selective checkpointing: ~8GB activation memory
• Recomputation overhead: ~20-25% increase in training step time This enables training on a single NVIDIA A100 (80GB) that would otherwise require model parallelism across multiple GPUs.

Not all operations benefit equally from checkpointing. Optimal strategies target high-activation, low-compute operations:

• Attention maps: Store QK^T matrices; they are memory-intensive but cheap to recompute
• Linear projections: Checkpoint aggressively; matrix multiplications are fast to recompute
• LayerNorm/GELU activations: Store these; they are computationally trivial but memory-light
• Convolutional stem layers: Avoid checkpointing; early feature maps are small and recomputation is expensive Frameworks like DeepSpeed and FairScale offer automatic activation partitioning to apply this selectively.

When combined with Automatic Mixed Precision (AMP) training, gradient checkpointing requires careful handling of precision casting. The recomputed forward pass must replicate the original precision state:

• FP16 forward: Recomputation occurs in FP16 to match the original activation values
• Master weights: Remain in FP32 and are not affected by checkpoint boundaries
• Loss scaling: The gradient scaling factor is applied identically to recomputed gradients This combination is standard in medical imaging pipelines using nnUNet or MONAI with ViT backbones, enabling batch sizes of 4-8 on 3D volumes that would otherwise be limited to batch size 1.

Gradient checkpointing is one of three primary memory-saving strategies, each with distinct trade-offs:

• Gradient Checkpointing: +25% compute, no CPU-GPU transfer overhead, deterministic
• Activation Offloading: Moves activations to CPU RAM; +10% compute but high PCIe bandwidth dependency
• Gradient Accumulation: Splits batch across micro-batches; no compute overhead but changes batch norm statistics For medical imaging with DICOM volumes, checkpointing is preferred over offloading when GPU compute is abundant but VRAM is constrained, as PCIe transfers become a bottleneck with high-resolution 3D data.

The formal name for the mechanism underlying gradient checkpointing. During the forward pass, only a subset of layer activations are retained in memory. During the backward pass, discarded activations are recomputed on-the-fly from the nearest saved checkpoint. This trades a ~20-30% increase in forward computation for a sublinear reduction in peak memory, enabling models like ViT-Large to train on consumer GPUs.

A complementary technique that stores activations and performs most arithmetic in 16-bit floating point (FP16 or BF16) rather than 32-bit. When combined with gradient checkpointing, the memory savings multiply: checkpointing reduces the number of stored tensors, while mixed precision halves the size of each remaining tensor. Modern frameworks like PyTorch's torch.cuda.amp implement this via automatic loss scaling to preserve numerical stability.

An I/O-aware exact attention algorithm that avoids materializing the full N×N attention matrix in GPU high-bandwidth memory (HBM). By tiling the computation and recomputing attention scores in fast on-chip SRAM during the backward pass, FlashAttention achieves the same memory-compute tradeoff as gradient checkpointing but specifically optimized for the self-attention bottleneck. Critical for training long-sequence vision transformers.

A strategy that splits a large logical batch into smaller micro-batches processed sequentially. Gradients are accumulated across micro-batches before a single optimizer step. This reduces peak activation memory linearly with the number of micro-batches. Combined with gradient checkpointing, it allows effective batch sizes that would otherwise exceed GPU memory by orders of magnitude, though at the cost of increased wall-clock training time.

When gradient checkpointing alone cannot fit a model, tensor parallelism shards individual layers across multiple GPUs, while pipeline parallelism partitions sequential layers across devices. Checkpointing integrates naturally with pipeline parallelism: only activations crossing pipeline stage boundaries must be communicated, and each stage can independently apply recomputation. Frameworks like Megatron-LM and DeepSpeed orchestrate this combination for trillion-parameter vision models.

An orthogonal technique that moves tensors from GPU memory to much larger CPU RAM during the forward pass and fetches them back during the backward pass. DeepSpeed ZeRO-Offload and PyTorch's torch.utils.checkpoint with use_reentrant=False support hybrid strategies where some activations are checkpointed and others are offloaded. The tradeoff is PCIe bandwidth: offloading adds latency, while recomputation adds compute. Optimal strategies balance both based on the GPU-CPU bandwidth ratio.