Gradient Checkpointing

The fundamental principle of gradient checkpointing is a time-memory trade-off. Instead of storing all intermediate activations from the forward pass—which consumes O(n) memory for n layers—the technique strategically discards most of them. During the backward pass, these discarded activations are recomputed on-demand from the nearest stored checkpoint. This reduces peak memory usage from O(n) to O(√n), enabling the training of models that are 2-4x larger on the same hardware, at the cost of approximately 30-40% more compute time for the extra forward passes.

Not all activations are checkpointed. The placement of checkpoints is a critical optimization problem. Common strategies include:

• Uniform Placement: Checkpointing every k-th layer (e.g., every 5 layers). Simple but suboptimal.
• Optimal Dynamic Programming: Algorithms that determine the checkpoint schedule to minimize total recomputation cost for a fixed memory budget, considering the varying computational cost of different layers.
• Manual Heuristics: Targeting memory-intensive layers (e.g., early transformer blocks with large sequence lengths). The goal is to store just enough activations to keep the recomputation cost manageable while staying within the memory limit.

Gradient checkpointing is implemented by intercepting the automatic differentiation (autodiff) engine. In PyTorch, this is done via torch.utils.checkpoint.checkpoint. The function wraps a segment of the model:

• Forward Pass: The wrapped segment runs, but its outputs are stored, not its internal activations. A custom autograd Function is recorded.
• Backward Pass: When gradients are requested, this custom function re-executes the wrapped segment in a no-grad context to recompute the internal activations, then performs the backward pass through the segment with gradients enabled. Frameworks like TensorFlow and JAX have similar APIs (tf.recompute_grad, jax.checkpoint).

The backward pass becomes a series of recomputation sweeps. Starting from the output and moving backward:

1. The system loads the nearest stored checkpoint activation.
2. It re-executes the forward pass for the subsequent non-checkpointed layers.
3. With the full set of activations now temporarily available for that segment, it computes the gradients for those layers.
4. The recomputed activations are discarded, and the process repeats for the next segment. This creates a wavefront of computation where memory is repeatedly allocated and freed for small segments, keeping the peak usage low. The process is automatic but adds significant computational overhead.

Gradient checkpointing is often used in conjunction with mixed precision training (using FP16/BF16). This combination is powerful but requires care:

• Checkpoint Precision: Typically, checkpoints are stored in FP32 (full precision) even when forward/backward passes use FP16. This prevents a buildup of quantization error during the chain of recomputations, which can degrade training stability.
• Recomputation Precision: The recomputation forward pass is usually performed in the same lower precision (FP16) as the original forward pass for speed.
• Memory Savings: Storing checkpoints in FP32 doubles their memory footprint compared to FP16, but this is offset by not storing all other activations. The net effect is still a substantial memory reduction.

Primary Use Cases:

• Training very large models (e.g., LLMs, diffusion models) on memory-constrained hardware.
• Processing extremely long sequences in transformers, where activation memory is the primary bottleneck.
• Enabling larger batch sizes for better hardware utilization and training stability.

Key Limitations:

• Compute Overhead: Adds 30-40% more FLOPs, increasing wall-clock training time.
• Implementation Complexity: Requires careful segmentation of the model; not all model architectures (e.g., those with complex control flow) are easily checkpointable.
• I/O Bottlenecks: On systems with slow CPU-GPU data transfer, the overhead can be higher. It is less beneficial for models where parameter memory, not activation memory, is the limiting factor.

A neural network architecture that routes different inputs to specialized sub-networks (experts). Instead of activating the entire model for every input, a gating network selects a small subset of experts. This enables a massive increase in model parameters (e.g., 1 trillion+) while keeping the computational cost per token similar to a much smaller dense model, as only a fraction of the total parameters are used.

An I/O-aware exact attention algorithm that dramatically speeds up transformer training and inference and reduces memory usage. Its key innovation is avoiding the materialization of the large, quadratic-sized attention matrix in slow High Bandwidth Memory (HBM). Instead, it recomputes attention scores on-the-fly within fast SRAM, performing tiling and recomputation to minimize memory reads/writes. This is a form of kernel fusion optimized for modern GPU memory hierarchy.

A process where a model is fine-tuned with simulated quantization noise applied during the forward pass. This allows the model weights to adapt to the lower precision (e.g., INT8) that will be used during deployment. Unlike post-training quantization, QAT accounts for the quantization error during training, typically recovering nearly all the accuracy lost by moving to lower bit-widths. It's a critical technique for on-device inference optimization.

A model compression technique that removes entire structural components from a neural network, such as neurons, channels, filters, or attention heads. Unlike unstructured pruning (which creates sparse weight matrices), structured pruning results in a smaller, denser architecture that can leverage standard, highly optimized dense linear algebra libraries. It directly reduces the model's parameter count and FLOPs, making it hardware-friendly.

A family of techniques where a model adapts its computational cost per input sample. The goal is to avoid wasting computation on easy examples. Key strategies include:

• Early Exiting: Attaching classifiers to intermediate layers, allowing simple inputs to exit early.
• Conditional Computation: Dynamically activating only a subset of the model's components (e.g., layers, experts) based on the input. This paradigm shifts from a fixed-cost to an input-adaptive cost model.

Empirically derived power-law relationships that predict how a model's pre-training loss scales with its size (parameters), dataset size, and computational budget. The Chinchilla scaling laws demonstrated that for a fixed compute budget, optimal performance is achieved by scaling model size and training data in roughly equal proportion, challenging prior assumptions. These laws are essential for compute-optimal planning of large-scale training runs.