Integer-Only Inference

The core principle of integer-only inference is the complete elimination of floating-point (FP) operations. All tensors—weights, activations, and intermediate results—are represented as integers. This allows the model to run on hardware lacking FP units (FPUs), such as many microcontrollers, low-power CPUs, and specialized neural processing units (NPUs) with integer-only cores. Operations like matrix multiplication and convolution are performed using integer multiply-accumulate (IMAC) instructions, which are faster and more energy-efficient than their FP counterparts on supported hardware.

In a standard quantized inference pipeline, integer tensors are often dequantized to floating-point for operations like batch normalization or activation functions, then re-quantized. Integer-only inference removes this costly back-and-forth. It achieves this by:

• Fusing scaling factors: The floating-point scale and zero-point parameters from quantization are fused into integer operations.
• Reformulating non-linearities: Functions like batch normalization, ReLU, and sigmoid are redesigned to use fixed-point integer arithmetic. For example, batch normalization is folded into preceding convolution weights, and sigmoid is approximated via integer-friendly polynomials or lookup tables (LUTs).

This paradigm delivers significant performance advantages on constrained hardware:

• Reduced Memory Bandwidth: Integer tensors (e.g., INT8) are 4x smaller than FP32, drastically cutting memory transfer costs.
• Faster Compute: Integer arithmetic units are simpler, smaller, and can achieve higher operation throughput per clock cycle and per watt compared to FP units.
• Deterministic Latency: Integer operations have fixed, predictable timing, unlike some FP operations, which is critical for real-time systems.
• Wider Hardware Support: Enables deployment on billions of existing edge devices with CPU cores that lack FPUs or have weak FP performance.

Since pure integers cannot represent fractions, integer-only inference relies on fixed-point arithmetic. A tensor is interpreted as an integer multiplied by a fixed, implicit scale factor (e.g., 2^-n).

• Scaling Factor Management: Each layer's operations must carefully manage the scale factors of inputs, weights, and outputs to prevent overflow and maintain precision.
• Bit-Shift Operations: Multiplication by a scale factor is often implemented as an efficient arithmetic bit-shift, avoiding general integer division.
• Accumulator Precision: Results of IMAC operations are stored in higher-bit-width integer accumulators (e.g., INT32) to prevent overflow before a final rescaling and casting back to lower precision (e.g., INT8).

Not all quantized models are ready for integer-only execution. Specific transformations are required during Quantization-Aware Training (QAT) or Post-Training Quantization (PTQ):

• Batch Norm Folding: Batch normalization layers are mathematically absorbed into the preceding convolutional or linear layer's weights and bias.
• Activation Function Quantization: All activation functions must have their input/output ranges precisely calibrated and replaced with integer-compatible approximations.
• Bias Correction: Biases are quantized to INT32 and added in the high-precision accumulator to maintain accuracy.
• Constant Folding: All scaling and zero-point calculations are pre-computed and baked into the model graph as constants.

Quantization-Aware Training (QAT) is a process where a neural network is trained or fine-tuned with simulated quantization operations embedded in the forward pass. This allows the model's parameters to adapt to the precision loss and distribution shifts caused by converting weights and activations to low-bit integers before deployment. It is a critical precursor to high-accuracy integer-only inference.

• Simulates Deployment Conditions: During training, floating-point values are quantized and de-quantized, mimicking the inference-time pipeline.
• Learns Robust Parameters: The model learns to compensate for quantization error, often outperforming models quantized after training (PTQ).
• Enables Integer-Only Graphs: QAT produces models where batch normalization can be fused and activation functions are replaced with integer-friendly approximations.

Post-Training Quantization (PTQ) is a compression technique that reduces the numerical precision of a pre-trained model's weights and activations to integers without requiring retraining. It uses a small calibration dataset to estimate the optimal dynamic range (scale and zero-point) for each tensor. PTQ is faster than QAT but may lead to higher accuracy degradation, especially for extreme quantization.

• Calibration is Key: A representative dataset is passed through the model to record activation ranges (e.g., using min/max or entropy methods).
• Static vs. Dynamic: Integer-only inference typically uses static quantization, where scale factors are fixed at compile time. Dynamic quantization calculates scales at runtime, adding overhead.
• Advanced Algorithms: Methods like AdaRound optimize weight rounding to minimize task loss, bridging the accuracy gap with QAT for low-bit models.

The Straight-Through Estimator (STE) is a gradient approximation method essential for training networks with non-differentiable, discrete-valued operations, such as quantization or binarization. During backpropagation, the STE treats the hard quantization function as if it had a gradient of 1, allowing gradients to flow through unchanged.

• Enables Gradient Flow: It bypasses the zero-gradient problem of step functions by defining a custom backward pass.
• Fundamental for Low-Bit Training: Techniques like BinaryConnect and the training of XNOR-Nets rely on the STE to learn binary or ternary weights.
• Can Introduce Bias: The gradient mismatch is an approximation that can lead to unstable training; it is often combined with techniques like gradient clipping.

Uniform quantization is the most common scheme for integer-only inference, where the continuous range of values is mapped to evenly spaced, discrete integer levels. It is defined by a scale (the distance between levels) and a zero-point (the integer value corresponding to real zero). This linearity is crucial for efficient integer arithmetic.

• Mathematical Simplicity: The quantization function is Q(r) = round(r / scale) + zero_point. Dequantization is r' = (Q(r) - zero_point) * scale.
• Enables Integer Math: A key property is that a matrix multiplication of quantized tensors can be performed using only integer operations, with the scale factors applied afterward.
• Contrast with Non-Uniform: Non-uniform quantization (e.g., logarithmic) can be more accurate for very low bits but often requires lookup tables or special hardware, breaking pure integer arithmetic.

In integer-only inference, a scaling factor (often denoted alpha or simply scale) is a per-tensor or per-channel floating-point value that recovers the dynamic range lost during integer quantization. It is the bridge between the efficient integer domain and the real-valued number system.

• Dequantization Multiplier: After an integer operation (e.g., convolution), the integer result is multiplied by the combined scale (weight_scale * input_scale) / output_scale to produce the correct floating-point output.
• Fused into Integer Operations: For full integer-only pipelines, this floating-point multiplication is often approximated by a fixed-point integer multiplication and bit-shift, or absorbed into adjacent layers.
• Granularity Matters: Layer-wise scaling uses one factor per tensor. Channel-wise scaling uses a unique factor per output channel, improving accuracy at the cost of more parameters.

Compute graph optimization is a compiler-level process that transforms a neural network's computational graph to enable efficient integer-only inference on target hardware. This involves fusing operations, rewriting layers, and eliminating data type conversions.

• Fusion and Rewriting: Critical optimizations include BatchNorm folding (absorbing batch normalization parameters into preceding convolution weights) and activation function fusion (replacing floating-point ReLU with integer-clipping).
• Quantization Node Placement: The compiler decides where to insert quantization (Q) and dequantization (DQ) nodes. For integer-only execution, Q/DQ nodes are pushed to the graph boundaries, and all internal operations use integer math.
• Hardware-Specific Kernels: The final graph is lowered to use highly optimized integer kernels (e.g., for ARM NEON or NPU accelerators) that avoid floating-point units entirely.