Rounding Mode (Quantization)

Round-to-Nearest (RTN), also called nearest rounding, is the most common deterministic rounding mode. It maps a floating-point value to the closest representable quantized level on the quantization grid. The tie-breaking rule for values exactly halfway between two levels is critical. The standard implementation is round-half-to-even (or banker's rounding), which rounds to the nearest even integer to avoid statistical bias. This mode is computationally simple and is the default in most Post-Training Quantization (PTQ) frameworks.

• Primary Use: General-purpose static and dynamic quantization.
• Key Property: Deterministic, minimizes mean absolute error for uniformly distributed data.
• Tie-breaking: Round-half-to-even prevents cumulative bias.

Stochastic Rounding is a non-deterministic mode that rounds a value up or down with a probability proportional to the distance to the adjacent quantization levels. For a value (x) between integers (i) and (i+1), it rounds to (i+1) with probability (p = x - i) and to (i) with probability (1-p).

• Primary Use: Quantization-Aware Training (QAT) and training low-precision models from scratch.
• Key Property: Unbiased estimator; the expected value of the rounded result equals the original floating-point value ((E[Q(x)] = x)).
• Benefit: Prevents gradient stagnation during low-bit training by providing a non-zero expected gradient through the rounding operation.

Round-Toward-Zero, often implemented as simple truncation, always rounds a value toward zero by discarding the fractional part. For positive numbers, this is floor rounding; for negative numbers, it is ceiling rounding. This mode introduces a consistent bias (negative for positive inputs, positive for negative inputs) because the rounded value is always smaller in magnitude than the original.

• Primary Use: Hardware implementations where truncation is a free byproduct of bit-shifting operations.
• Key Property: Fast but biased; not typically used for activation quantization due to error accumulation.
• Example: The floating-point value 2.7 is truncated to 2, and -2.7 is truncated to -2.

Floor Rounding (round down) and Ceiling Rounding (round up) are deterministic modes that always round to the next lowest or highest quantized level, respectively. Floor rounding applies the mathematical floor() function, while ceiling rounding applies ceil().

• Floor Use Cases: Guaranteeing upper bounds in safety-critical systems or specific hardware instructions.
• Ceiling Use Cases: Ensuring minimum resource allocation or meeting latency guarantees.
• Key Property: Both introduce significant directional bias. They are rarely used for general model quantization but can be components of more advanced schemes or used in specific layers to control error direction.

When a value is exactly midway between two quantized levels, Round-to-Nearest (RTN) requires a tie-breaking rule. The choice influences long-term statistical error.

• Round-Half-Up: Rounds away from zero (e.g., 2.5 → 3, -2.5 → -3). Introduces a positive bias for positive numbers.
• Round-Half-Down: Rounds toward zero (e.g., 2.5 → 2, -2.5 → -2). Introduces a negative bias for positive numbers.
• Round-Half-To-Even (Banker's Rounding): Rounds to the nearest even number (e.g., 2.5 → 2, 3.5 → 4). This is the standard in IEEE 754 and most ML frameworks (like PyTorch's torch.round). It minimizes the expected average error over many samples by alternating the direction of rounded ties, preventing cumulative bias.

The choice of rounding mode is constrained by and optimized for target hardware. Most dedicated AI accelerators and Neural Processing Units (NPUs) natively support only specific rounding logic in their integer arithmetic units.

• Deployment Standard: Round-to-Nearest (RTN) with round-half-to-even is the ubiquitous default for Post-Training Quantization (PTQ) and inference runtimes (e.g., TFLite, ONNX Runtime).
• Training Requirement: Stochastic Rounding is essential for Quantization-Aware Training (QAT) and training ultra-low precision (e.g., 4-bit) models, as it provides unbiased gradients.
• Performance: Truncation or floor rounding may be used internally in hardware for speed, with software compensating for the bias elsewhere in the pipeline. The selected mode must be consistent between the calibration/quantization tool and the hardware's low-level kernels.

Quantization error is the numerical distortion introduced when mapping a continuous floating-point value to a discrete quantized level. It is the fundamental trade-off in quantization, directly influenced by the rounding mode.

• Types of Error: Includes rounding error (from the mapping rule) and clipping/saturation error (from values outside the representable range).
• Measurement: Often quantified as the difference between the original floating-point tensor and its dequantized counterpart, using metrics like Mean Squared Error (MSE).
• Impact: The choice of rounding mode (e.g., round-to-nearest vs. stochastic) is a primary method for managing and distributing this error.

The quantization grid is the finite set of discrete integer values that represent the allowable output levels for a quantized tensor, defined by the bit-width and range.

• Structure: For uniform quantization, the grid consists of evenly spaced levels (e.g., for 8-bit: -128 to 127). The rounding mode determines which grid point a given float value is assigned to.
• Granularity: The density of the grid is determined by the bit-width; lower bit-width means a coarser grid, making the rounding decision more critical.
• Interaction with Rounding: Modes like round-to-nearest-ties-to-even explicitly define behavior when a value is exactly halfway between two grid points.

Stochastic rounding is a probabilistic rounding mode where a floating-point value is rounded up or down with a probability proportional to its distance to the two nearest quantization levels.

• Mechanism: A value of 2.7 on an integer grid might round to 3 with 70% probability and to 2 with 30% probability.
• Use Case: Primarily used during Quantization-Aware Training (QAT). By making the rounding operation stochastic, it becomes differentiable in expectation, allowing gradients to flow and the model to adapt to quantization noise.
• Benefit: Helps prevent the convergence issues and accuracy loss that can occur with deterministic rounding during training, often leading to more robust final quantized models.

Round-to-nearest (RTN) is the most common deterministic rounding mode, mapping a floating-point value to the closest representable quantized level.

• Default for Inference: It is the standard mode for Post-Training Quantization (PTQ) and deployed models due to its simplicity and deterministic behavior.
• Tie-Breaking: Requires a tie-breaking rule for values exactly midway between two levels. Round-to-nearest-ties-to-even (also known as "bankers' rounding") is a common variant that minimizes statistical bias.
• Hardware Support: Directly implemented in most integer arithmetic logic units (ALUs) and AI accelerators, making it highly efficient for inference.

Quantization-Aware Training (QAT) is a process where quantization effects, including the rounding operation, are simulated during the training or fine-tuning phase.

• Role of Rounding: QAT frameworks use a Straight-Through Estimator (STE) to approximate the gradient of the non-differentiable rounding function, allowing the model weights to be optimized for the low-precision target.
• Mode Selection: QAT often employs stochastic rounding during the forward pass to improve training dynamics, but typically switches to deterministic round-to-nearest for the final model export.
• Outcome: Produces models that maintain significantly higher accuracy after quantization compared to standard Post-Training Quantization (PTQ).

Integer arithmetic refers to the low-precision mathematical operations performed on quantized values, which is the primary efficiency gain from quantization.

• Hardware Acceleration: Modern CPUs, GPUs, and Neural Processing Units (NPUs) have specialized integer units (e.g., INT8, INT4) that execute these operations much faster and with lower power consumption than floating-point units.
• Rounding Requirement: The affine quantization transformation (scale * (int_val - zero_point)) must result in an integer. The rounding mode determines how the result of intermediate calculations is mapped back to the integer grid before the next operation.
• Stack Dependency: The choice of rounding mode must be consistent across the training framework, model converter (e.g., TFLite), and target hardware runtime to ensure bit-accurate results.