FP16 Quantization

FP16 adheres to the IEEE 754-2008 binary16 standard, allocating 1 sign bit, 5 exponent bits, and 10 mantissa bits. This dynamic range (≈ 6×10⁻⁸ to 65,504) preserves both very small gradients and large activations, unlike integer quantization formats. The exponent bias of 15 enables representation of subnormal numbers, preventing underflow to zero during backpropagation.

Modern frameworks implement automatic mixed precision (AMP) rather than pure FP16 execution. Key mechanics:

• Weights: Stored in FP32 master copy for stability
• Forward/backward passes: Computed in FP16 for speed
• Loss scaling: Multiplied by a dynamic factor before backpropagation to prevent gradient underflow below 2⁻²⁴
• Weight updates: Applied in FP32 to preserve small-magnitude adjustments This hybrid approach achieves near-identical convergence to full FP32 training.

Quantizing from FP32 to FP16 reduces tensor memory footprint by exactly 50%. For a ResNet-50 with 25.6M parameters:

• FP32 model: 102.4 MB
• FP16 model: 51.2 MB This doubles the effective batch size that fits in GPU VRAM and halves the time spent on memory-bound operations. On memory-bandwidth-constrained edge NPUs, this directly translates to 2× inference throughput.

FP16 throughput gains require explicit hardware support via tensor cores or dedicated FP16 ALUs:

• NVIDIA: Tensor Cores on V100 (125 TFLOPS FP16), A100 (312 TFLOPS FP16)
• Apple: ANE supports FP16 natively on A-series and M-series chips
• Qualcomm: Hexagon NPU with FP16 vector extensions
• Intel: XMX engines on Arc GPUs and Gaudi accelerators Without these units, FP16 execution falls back to FP32 emulation, negating performance benefits.

For deployment scenarios where post-training quantization causes unacceptable accuracy degradation, QAT simulates FP16 quantization during the forward pass while maintaining FP32 gradients:

• Inserts fake quantization nodes that round values to FP16 precision
• Backpropagation uses straight-through estimator (STE) to bypass non-differentiable rounding
• Model learns to adapt weights to the reduced precision, minimizing accuracy loss QAT typically recovers 0.5-2% accuracy compared to naive post-training FP16 conversion.

FP16's limited range (max 65,504) introduces specific failure modes:

• Overflow: Activations exceeding 65,504 saturate to infinity, propagating NaN values
• Underflow: Gradients below 6×10⁻⁸ flush to zero, stalling learning
• Precision loss: Values near 1.0 have only ~0.001 resolution, causing quantization noise in softmax and layer normalization Mitigation strategies include gradient clipping, loss scaling factors (typically 2⁸ to 2¹⁶), and keeping sensitive operations like batch norm statistics in FP32.

A more aggressive compression technique that maps 32-bit floating-point values to 8-bit integers. While FP16 maintains a dynamic range suitable for both weights and activations, INT8 requires calibration to determine optimal clipping ranges. It delivers roughly 2x further memory reduction compared to FP16 but often necessitates quantization-aware training (QAT) to recover accuracy lost from the lower bit depth.

A training paradigm that uses FP16 for forward and backward passes to accelerate computation while maintaining a master copy of weights in FP32 to preserve numerical stability during updates. This approach, popularized by NVIDIA's Apex library, prevents gradient underflow—a common failure mode where small gradient values become zero in half-precision, stalling learning entirely.

A truncated 32-bit float format that retains the same 8-bit exponent range as FP32 but reduces the mantissa to 7 bits. Unlike FP16, BF16 sacrifices fractional precision to preserve dynamic range, making it less prone to overflow/underflow during training. It is natively supported on Google TPUs and Intel Habana accelerators and often requires no loss scaling.

A one-shot compression technique applied to a fully trained FP32 model without retraining. FP16 quantization is the simplest form of PTQ—weights are directly cast to half-precision. More complex PTQ methods for INT8 involve:- Calibration datasets to observe activation ranges- Layer fusion (merging Conv + BatchNorm + ReLU)- Bias correction to compensate for quantization errorPTQ is ideal when training data is inaccessible or retraining is cost-prohibitive.

A technique required during FP16 training to prevent small gradient values from underflowing to zero. The loss value is multiplied by a large constant (e.g., 1024) before backpropagation, shifting gradient magnitudes into the representable FP16 range. The weights are then unscaled before the optimizer update. Modern frameworks implement dynamic loss scaling, which automatically adjusts the factor to prevent overflow.

A complementary compression technique that removes redundant weights or neurons by setting them to zero, creating sparse tensors. When combined with FP16 quantization, pruned models achieve compound compression: sparsity reduces the number of operations, while half-precision reduces the size of each remaining element. Structured pruning, which removes entire channels, is particularly effective for maintaining hardware-friendly memory access patterns.