Mixed Precision Training

A full-precision FP32 master copy of all model weights is maintained throughout training. During each iteration, weights are cast to FP16 for the forward and backward passes to leverage faster half-precision compute. The resulting FP16 gradients are then used to update the master FP32 weights, preventing the gradual loss of small-magnitude updates that would otherwise vanish in lower precision.

A technique to preserve gradient values that underflow to zero in FP16. The loss is multiplied by a large scaling factor before backpropagation, shifting small gradient magnitudes into the representable FP16 range. After the backward pass, gradients are unscaled by the same factor before the optimizer step. Modern frameworks implement dynamic loss scaling, which automatically adjusts the scale factor by monitoring gradient overflow to find the optimal range.

NVIDIA Tensor Cores are specialized hardware units that perform fused multiply-add operations on FP16 matrices, delivering up to 8x higher throughput compared to standard FP32 CUDA cores. Mixed precision training is explicitly designed to exploit these units by ensuring matrix multiplications and convolutions execute in half-precision, while accumulation steps retain FP32 precision to maintain numerical stability.

Storing activations and gradients in FP16 halves the memory footprint of these tensors compared to FP32. This enables:

• Larger batch sizes that improve training stability
• Deeper or wider models that fit within GPU memory constraints
• Reduced I/O pressure on memory bandwidth-bound operations For large vision transformers, this often translates to a 40-50% reduction in total GPU memory consumption.

Mixed precision shifts the bottleneck from compute-bound to memory-bandwidth-bound operations. By reducing the byte count of every tensor transfer, operations like element-wise additions and normalization layers execute faster. This is particularly impactful for Vision Transformers, where the self-attention mechanism involves numerous memory-intensive reshaping and scaling operations between the compute-heavy matrix multiplications.

Two primary implementation paths exist:

• NVIDIA Apex: A PyTorch extension offering opt_level presets (O0-O3) that progressively automate mixed precision, from conservative FP32 fallback lists to aggressive full-FP16 execution
• Native Torch AMP: torch.cuda.amp provides context managers (autocast) and gradient scalers (GradScaler) with fine-grained control, automatically casting eligible ops to FP16 while keeping safety-critical ops in FP32

A software framework that automates the casting of operations to lower precision. AMP automatically selects the appropriate data type for each operation, using FP16 for matrix multiplications and convolutions while keeping reductions and normalizations in FP32. In PyTorch, this is implemented via torch.cuda.amp with a GradScaler that dynamically scales the loss to prevent gradient underflow in the 16-bit range.

A critical technique that multiplies the loss value by a large constant before backpropagation to push small gradient values into the representable range of FP16. Without scaling, gradients for many parameters would underflow to zero, stalling training. Modern implementations use dynamic loss scaling, which automatically increases or decreases the scale factor based on a history of gradient overflow checks.

A 19-bit data format native to the NVIDIA Ampere architecture and beyond. TF32 uses the same 10-bit mantissa as FP16 but retains the 8-bit exponent range of FP32, providing the dynamic range of 32-bit with the throughput of 16-bit. On A100 and H100 GPUs, TF32 is the default math mode for matrix multiplications, offering a 1.6x speedup over FP32 with no code changes required.

A 16-bit floating-point format with the same 8-bit exponent as FP32 but a truncated 7-bit mantissa. Unlike FP16, BF16 preserves the full dynamic range of FP32, eliminating the need for loss scaling. This format is particularly advantageous for medical imaging models where large activation ranges are common. BF16 is natively supported on Google TPUs and NVIDIA A100/H100 GPUs.

The next frontier in low-precision training, using 8-bit floating-point formats on NVIDIA H100 GPUs. FP8 comes in two variants: E4M3 (4 exponent bits, 3 mantissa bits) for forward propagation and E5M2 (5 exponent bits, 2 mantissa bits) for gradients. This requires fine-grained scaling factors computed per-tensor or per-block to maintain numerical stability, enabling another 2x throughput improvement over FP16.

The foundational mechanism that makes mixed precision safe. A full-precision FP32 master copy of all model weights is maintained throughout training. Each iteration, a truncated FP16 copy is created for the forward and backward passes to maximize speed. The resulting FP16 gradients are then used to update the FP32 master weights, preserving the precision necessary for stable convergence over many iterations.