FP16 (Half-Precision)

The FP16 format allocates its 16 bits as follows: 1 sign bit, 5 exponent bits, and 10 mantissa bits (also called the significand or fraction). This structure provides approximately 3.31 decimal digits of precision. The 5-bit exponent allows for a representable range from approximately 6.1 × 10⁻⁵ (2⁻¹⁴) to 65504 (2¹⁶). The limited mantissa bits are the primary source of quantization error, as values must be rounded to the nearest representable number.

A key distinction is FP16's dynamic range relative to BFloat16 (BF16). While both are 16-bit, BF16 uses an 8-bit exponent (matching FP32) and a 7-bit mantissa. This gives BF16 the same dynamic range as FP32 but lower precision. FP16's 5-bit exponent creates a much narrower representable range. This makes FP16 susceptible to numerical underflow, where small gradient values in training can become zero, and overflow, where large values become infinite. BF16 is often preferred for training for this stability.

The primary advantage of FP16 is its efficient use of memory and bandwidth. Compared to standard 32-bit floating-point (FP32):

• 2x Reduction in Model Size: Storing weights in FP16 halves the memory required.
• 2x Reduction in Memory Bandwidth: Transferring tensors between memory and compute units is twice as fast.
• Increased Cache Efficiency: More weights/activations can fit into the same size of high-speed cache (L1, L2, SRAM). This directly translates to higher theoretical throughput and lower latency for memory-bound operations, a critical factor in large model inference.

Modern GPUs (e.g., NVIDIA Volta architecture and later) and other AI accelerators feature specialized units for FP16 arithmetic. NVIDIA's Tensor Cores, for example, are designed to perform mixed-precision matrix multiply-accumulate operations, such as D = A * B + C, where A and B are FP16 matrices, while C and D can be FP16 or FP32. These cores provide a massive throughput advantage—up to 16x higher peak FLOPS for FP16 versus FP32 on the same hardware—when computations are properly structured to utilize them.

In inference scenarios, FP16 is a cornerstone of mixed precision inference. Common patterns include:

• Storing the model weights in FP16.
• Performing the bulk of matrix multiplications and convolutions in FP16.
• Using FP32 or higher precision for reduction operations, softmax, or layer normalization to maintain numerical stability. This approach, often automated by frameworks like PyTorch AMP or TensorFlow Mixed Precision, can yield 1.5x to 3x inference speedups on supported hardware with minimal accuracy loss for many models.

When used in training, FP16's limited range requires techniques to maintain stability. The most critical is loss scaling. Because gradient values can underflow to zero in FP16, the training loss is multiplied by a large scale factor (e.g., 128, 1024) before backpropagation. This shifts gradient values into FP16's representable range. The gradients are then unscaled before the optimizer updates the weights in a master FP32 copy. This simple technique prevents gradient underflow and is a standard component of automatic mixed precision (AMP) training pipelines.

The primary use case for FP16 is to accelerate inference on hardware with dedicated low-precision units. NVIDIA Tensor Cores (Volta architecture and later) and AMD Matrix Cores (CDNA architecture) perform matrix multiplications in FP16 at significantly higher throughput than FP32. This reduces latency and increases throughput for transformer-based models (LLMs, vision transformers) and convolutional networks. Key benefits include:

• 2-4x theoretical speedup for matrix-heavy ops.
• Halved memory bandwidth and storage requirements for weights and activations.
• Enables larger batch sizes or longer context windows within fixed GPU memory.

FP16's limited dynamic range (≈ ±65,504) and precision (10-bit mantissa) mandate careful application. It is generally safe for:

• Weight storage and computation in well-conditioned models.
• Activation functions with bounded ranges (e.g., ReLU, GELU). Risks and mitigations include:
• Gradient/Activation Underflow: Values < ~6e-8 become zero. Mitigated via loss scaling in training; for inference, monitoring for vanishing activations.
• Overflow in Attention Scores: Large dot products in transformers can overflow. Mitigated by scaling attention scores (standard practice) or casting sensitive ops to FP32.
• Accumulation in FP32: Best practice is to compute dot products in FP16 but accumulate in FP32, which hardware often does internally.

Not all hardware treats FP16 equally. Key distinctions:

• NVIDIA Tensor Cores: Perform FP16 matrix multiply with FP32 accumulation (HMMA instruction), combining speed with numerical safety. Pure FP16 math is also available.
• AMD CDNA/ROCm: Similarly supports FP16 via Matrix Cores, with framework support through ROCm and libraries like MIOpen.
• Intel Habana Gaudi: Uses BF16 as primary low-precision format; FP16 support varies.
• Apple Neural Engine: Prefers 16-bit floating-point (often float16).
• CPU Inference: x86 AVX-512 includes FP16 instructions (AVX512-FP16), but speedups are less dramatic than on GPUs. The decision to use FP16 must be validated per target platform.

Numerical stability is the challenge of maintaining model accuracy when using reduced precision. For FP16, the primary risks are:

• Underflow: Gradient values smaller than ~5.96e-8 become zero.
• Overflow: Values larger than 65504 become infinite.

Loss scaling is a critical technique to combat underflow. It involves:

1. Multiplying the loss value by a scale factor (e.g., 1024) before backpropagation.
2. This shifts gradient values into the FP16 representable range.
3. Gradients are unscaled before the optimizer updates the FP32 master weights.

INT8 quantization is a more aggressive precision reduction technique that represents model weights and activations using 8-bit integers. Compared to FP16, it offers:

• 4x memory reduction vs. FP32 (INT8) vs. 2x (FP16).
• Faster computation on hardware with integer ALUs.
• Requires a calibration step to map float ranges to integer ranges.
• Often used in conjunction with FP16 in mixed-precision systems, where certain layers (e.g., embeddings) remain in FP16 for stability, while compute-intensive layers (e.g., matrix multiplies) use INT8.

Model casting, or precision casting, is the explicit conversion of tensors from one numerical data type to another within a computational graph. It is a fundamental operation in mixed precision workflows. Examples include:

• Casting FP32 model weights to FP16 before a matrix multiplication.
• Casting FP16 activations back to FP32 for a sensitive accumulation operation.
• Strategic casting is required to avoid type mismatch errors and to place operations where they are hardware-optimized (e.g., keeping certain ops in FP32 for accuracy).