Int8 Inference

Int8 quantization maps 32-bit floating-point weights and activations to 8-bit integers, reducing memory bandwidth by 4x and enabling integer-only compute paths. The quantization formula q = round(x / scale + zero_point) introduces a small quantization error—typically less than 0.5% accuracy degradation on well-calibrated models. This trade-off is acceptable for most medical diagnostic models where latency requirements are strict.

• Memory reduction: 4x smaller model footprint
• Speedup: 2-4x faster inference on CPUs with SIMD instructions
• Accuracy impact: <1% top-1 accuracy loss with proper calibration

Symmetric quantization maps floating-point values to a signed integer range centered at zero, using a single scale factor. This simplifies hardware implementation but wastes representation on negative values for ReLU-based activations. Asymmetric quantization uses both a scale and a zero-point offset, fully utilizing the [0, 255] range for unsigned integers.

• Symmetric: q = round(x / scale), zero-point fixed at 0
• Asymmetric: q = round(x / scale) + zero_point, better for ReLU outputs
• Most edge NPUs prefer symmetric for weights, asymmetric for activations

QAT simulates quantization effects during the forward pass of training by inserting fake quantization nodes that round values to Int8 precision. The backward pass uses straight-through estimation to propagate gradients through the non-differentiable rounding operation. This allows the model to learn parameters robust to quantization error.

• Fake quantization: Simulates Int8 rounding during training
• Straight-through estimator: Approximates gradient through rounding
• Produces 1-3% higher accuracy than post-training quantization
• Essential for low-bit (4-bit, 2-bit) quantization schemes

Per-tensor quantization applies a single scale and zero-point to an entire weight tensor, which is computationally efficient but fails when different channels have widely varying value ranges. Per-channel quantization assigns separate quantization parameters to each output channel, preserving more information at the cost of slightly more complex dequantization logic.

• Per-tensor: One (scale, zero_point) per tensor, faster on generic CPUs
• Per-channel: One (scale, zero_point) per channel, preserves accuracy
• Per-channel is standard for weight quantization in TFLite and ONNX Runtime
• Activations typically use per-tensor quantization due to dynamic ranges

Before Int8 inference, a calibration dataset—a small, representative sample of unlabeled input data—is run through the model to observe activation value distributions. The calibration process determines optimal clipping ranges by minimizing information loss, using algorithms like KL divergence minimization or mean squared error minimization.

• Calibration dataset: 100-500 representative samples
• Methods: MinMax, MovingAverageMinMax, KL divergence, MSE
• Poor calibration causes clipping errors and accuracy collapse
• Critical for medical imaging models with outlier pixel intensities

Modern edge processors provide dedicated SIMD instructions for Int8 matrix multiplication. ARM Cortex-A CPUs use NEON dot-product instructions (SDOT, UDOT) that compute four 8-bit multiplications and accumulate into a 32-bit result in a single cycle. Intel CPUs leverage VNNI (Vector Neural Network Instructions) for Int8 convolution acceleration.

• ARM NEON: SDOT/UDOT for 4-way Int8 dot products
• Intel VNNI: VPDPBUSD for Int8 deep learning acceleration
• NPUs: Native Int8 tensor cores with 10-100x efficiency vs FP32
• Medical wearables benefit from 10x energy reduction with Int8 NPU execution

The parent compression technique that reduces the numerical precision of a neural network's weights and activations from 32-bit floating-point (FP32) to lower-bit representations. Int8 is the most widely supported quantization scheme, offering a 4x reduction in model size and significant speedup on CPUs and edge accelerators. Quantization can be applied post-training or during quantization-aware training (QAT) to recover accuracy lost through precision reduction.

A post-training quantization method where weights are statically quantized to Int8 at conversion time, while activation ranges are dynamically calculated during inference. This approach requires no calibration dataset and is the simplest path to Int8 deployment. However, dynamic range quantization introduces runtime overhead for activation scaling, making it slightly slower than full integer quantization on edge accelerators.

A small, representative sample of unlabeled data used to determine optimal clipping ranges and scaling factors for activations during static quantization. The calibration dataset must capture the statistical distribution of real inference inputs. Common calibration methods include:

• MinMax: Uses the absolute min and max values observed
• MovingAverageMinMax: Tracks exponential moving averages
• Histogram: Uses entropy or percentile-based range selection

A graph optimization that combines multiple discrete operations into a single computational kernel before quantization. For Int8 inference, fusing a convolution, batch normalization, and ReLU activation into one operation eliminates intermediate memory writes and reduces quantization error propagation. This technique is critical for maximizing throughput on edge NPUs and DSPs with limited memory bandwidth.

A training methodology that simulates quantization effects during the forward pass by inserting fake quantization nodes. The model learns to compensate for precision loss, producing weights robust to Int8 rounding errors. QAT consistently achieves higher accuracy than post-training quantization, especially for models below 8 bits or those with sensitive layers like attention mechanisms in transformers.

Two granularity strategies for computing quantization parameters:

• Per-Tensor: A single scale and zero-point for an entire weight tensor. Faster but less accurate for weights with high variance across channels.
• Per-Channel: Separate scale and zero-point for each output channel. Standard for Int8 convolution weights as it preserves accuracy with minimal overhead. Most edge NPUs require per-channel quantization for weights and per-tensor for activations.