Mixed Precision Inference

Uses 16-bit floating-point format for compute-intensive operations like matrix multiplications and convolutions. FP16 halves memory bandwidth requirements and enables faster tensor core operations on modern GPUs. However, its limited dynamic range can cause gradient underflow during training. For inference, the primary risk is accuracy degradation in networks with wide value distributions.

• Memory footprint: 2 bytes per value
• Dynamic range: ~5.96×10⁻⁸ to 65,504
• Best for: Convolutional layers, attention mechanisms
• Risk: Underflow for values below 6×10⁻⁸

A 16-bit format that preserves the 8-bit exponent of FP32 while truncating the mantissa to 7 bits. This design choice means BF16 has the same dynamic range as FP32 (~3.4×10³⁸), eliminating underflow and overflow concerns that plague FP16. The trade-off is reduced fractional precision.

• Memory footprint: 2 bytes per value
• Dynamic range: Identical to FP32
• Best for: Transformer models, large language models
• Advantage: No loss scaling required during training

A critical architectural pattern where a full-precision FP32 copy of model weights is maintained in memory, even when computations use lower precision. During each forward pass, weights are cast down to FP16/BF16 for computation, but the master copy preserves small weight updates that would otherwise be lost to quantization error.

• Storage overhead: 2x the quantized weights
• Purpose: Prevent accuracy drift over many inference cycles
• Common in: Mixed precision training and fine-tuning
• Not always needed: Pure inference with static weights can omit this

Reduces weights and activations to 8-bit integers for maximum throughput on hardware with INT8 tensor cores. Requires calibration using a representative dataset to determine optimal scaling factors that map floating-point ranges to integer buckets.

• Memory footprint: 1 byte per value
• Throughput: Up to 4x vs FP32 on supporting hardware
• Calibration methods: Min-max, percentile, KL divergence
• Risk: Accuracy loss requires per-layer validation
• Tools: NVIDIA TensorRT, ONNX Runtime, OpenVINO

A technique to combat gradient underflow when using FP16 for training. Small gradient values that fall below FP16's minimum representable value (~6×10⁻⁸) become zero, stalling learning. Loss scaling multiplies the loss by a large constant before backpropagation, shifting gradients into the representable range, then unscales them before the weight update.

• Static scaling: Fixed multiplier (e.g., 1024)
• Dynamic scaling: Automatically adjusts based on gradient statistics
• Inference relevance: Not needed for pure forward passes
• Framework support: Native in PyTorch AMP, TensorFlow

The process of mapping continuous 32-bit floating-point values to discrete lower-bit representations, typically INT8 or INT4. Unlike mixed precision which retains FP32 master weights, quantization permanently reduces model precision to shrink memory footprint and accelerate inference on integer-optimized hardware.

• Post-Training Quantization (PTQ): Calibrates scaling factors after training without retraining
• Quantization-Aware Training (QAT): Simulates quantization noise during training for higher accuracy
• Reduces model size by 2-4x with minimal accuracy loss

A 16-bit floating-point format that preserves the 8-bit exponent range of FP32 while truncating the mantissa to 7 bits. This design makes BF16 ideal for mixed precision training and inference because it maintains dynamic range—preventing overflow and underflow—while halving memory bandwidth requirements.

• Same exponent range as FP32: handles values from ~10^-38 to ~10^38
• No loss scaling required, unlike FP16
• Native support on NVIDIA A100/H100, AMD MI250, and Google TPU v4+

A framework-level feature in PyTorch and TensorFlow that automates the selection of FP16 or BF16 for eligible operations while retaining FP32 for precision-critical computations. AMP inserts cast operations transparently, allowing developers to enable mixed precision with minimal code changes.

• PyTorch: torch.cuda.amp.autocast() context manager
• TensorFlow: tf.keras.mixed_precision policy API
• Typically yields 1.5-3x training speedup on modern GPUs

A technique required for FP16 mixed precision to prevent gradient underflow. Small gradient values that fall below the minimum FP16 representable range (~6×10^-8) become zero, stalling training. Loss scaling multiplies the loss by a large factor before backpropagation, then unscales gradients before the optimizer step.

• Dynamic loss scaling automatically adjusts the scale factor during training
• Not required for BF16 due to its wider exponent range
• Critical for networks with many normalization layers where gradients are inherently small