Mixed-Precision Inference

The most common mixed-precision paradigm pairs FP32 master weights with FP16 compute activations. Matrix multiplications and convolutions execute in half-precision to double throughput on tensor cores, while accumulation and sensitive operations remain in single precision to prevent gradient underflow. This balances the 2x memory bandwidth reduction of FP16 against the numerical stability of FP32.

For deployment on edge accelerators, weights and activations are quantized to 8-bit integers. This reduces model size by 4x compared to FP32 and enables high-throughput integer math on CPUs and dedicated NPUs. Calibration is required to determine optimal clipping ranges and scale factors, mapping the dynamic range of floating-point tensors to the limited integer representation without significant accuracy degradation.

A critical technique enabling stable FP16 training and fine-tuning. Gradients in half-precision often underflow to zero due to limited dynamic range. Loss scaling multiplies the loss value by a large constant before backpropagation, shifting small gradient values into the representable FP16 range. The gradients are then unscaled before the weight update. Modern frameworks implement dynamic loss scaling that automatically adjusts the factor to prevent overflow.

Advanced compilers analyze the sensitivity of each layer to quantization error and assign precision accordingly. Layers with high activation variance or outlier weights retain higher precision (FP16 or INT16), while robust layers drop to INT8 or INT4. This per-layer or per-channel granularity is a key feature of toolchains like TensorRT and Vitis AI, which profile the model on target silicon to generate an optimal precision schedule.

BFloat16 (BF16) is a 16-bit floating-point format that retains the 8-bit exponent of FP32 but truncates the mantissa to 7 bits. This preserves the wide dynamic range of FP32, eliminating the need for loss scaling, while still halving memory bandwidth. BF16 is the preferred format for training and inference on Google TPUs and Intel Cooper Lake CPUs, trading some fractional precision for robust numerical stability.

When quantizing weights or activations, standard nearest-even rounding introduces a systematic bias that degrades model accuracy over many operations. Stochastic rounding probabilistically rounds a value up or down based on its fractional component, preserving the expected value of the distribution. This is crucial for INT8 quantization-aware training, where unbiased gradient accumulation prevents accuracy collapse in deep networks.