Model Quantization

Quantization reduces the bit-width of a neural network's weights and activations. The standard baseline is FP32 (32-bit floating point), which is reduced to INT8 (8-bit integer) or even INT4 for extreme compression. This conversion replaces continuous floating-point values with discrete integer representations using a scale factor and zero point. The primary benefit is a theoretical 4x reduction in model size and memory bandwidth requirements, enabling deployment on microcontrollers and medical wearables with limited SRAM.

Quantization-Aware Training simulates the effects of low-precision arithmetic during the forward and backward passes of model training. The model maintains full-precision shadow weights while inserting fake quantization nodes that round values to simulate INT8 behavior. This allows the optimizer to learn parameters that are robust to quantization error. QAT consistently achieves higher accuracy than post-training methods, especially for models with depthwise separable convolutions or efficient attention mechanisms common in mobile vision and on-device NLP models.

Post-Training Quantization converts a pre-trained FP32 model to INT8 without retraining. A small calibration dataset of unlabeled samples is passed through the model to collect activation statistics and determine optimal clipping ranges. Common PTQ schemes include:

• Dynamic Range Quantization: Weights are statically quantized; activations are quantized dynamically at runtime.
• Full Integer Quantization: Both weights and activations are statically quantized, required for hardware accelerators that lack floating-point units. PTQ is ideal when training pipelines or full datasets are unavailable.

Quantization granularity determines the scope of the scale factor and zero point parameters:

• Per-Tensor Quantization: A single scale and zero point for an entire weight tensor. Simple but suffers when weight distributions vary significantly across channels.
• Per-Channel Quantization: Independent scale factors for each output channel of a convolutional or linear layer. This preserves accuracy better for models with batch normalization layers where channel-wise variance is high. Modern edge NPUs and DSPs increasingly support per-channel INT8 operations natively.

The mapping function from floating-point to integer defines the quantization scheme:

• Symmetric Quantization: Maps the floating-point range symmetrically around zero, using only a scale factor. The zero point is fixed at 0. Efficient for weights that are roughly zero-centered.
• Asymmetric Quantization: Uses both a scale factor and a non-zero zero point to map the exact min/max range. Better for ReLU activations which are strictly non-negative, as it avoids wasting integer range on negative values. The choice impacts both accuracy and the complexity of integer matrix multiplication kernels.

Quantized models achieve real speedups only when the target hardware has native INT8 SIMD instructions or dedicated matrix engines. Key considerations:

• ARM Cortex-M: Requires CMSIS-NN kernels optimized for INT8 convolutions.
• Qualcomm Hexagon DSP: Supports INT8 operations via the Hexagon Tensor Processor.
• Apple Neural Engine: Natively accelerates INT8 and FP16 operations on A-series and M-series chips.
• Operator Coverage: Not all operations quantize efficiently. LSTM cells, softmax, and layer normalization often require higher precision or custom quantized implementations to avoid significant error accumulation.

The execution of a neural network using 8-bit integer arithmetic instead of 32-bit floating-point. This is the most common quantization target, providing a 4x reduction in model size and significant speedup on CPUs and edge accelerators. Key characteristics:

• Weights and activations are represented as INT8 values
• Requires a calibration dataset to determine optimal scaling factors
• Typically achieves < 1% accuracy loss for well-optimized models
• Supported natively by most edge inference runtimes including ONNX Runtime and TensorFlow Lite

A post-training quantization method that statically quantizes only the model weights to 8-bit integers while dynamically calculating quantization ranges for activations at runtime. This approach:

• Requires no calibration dataset for activation ranges
• Offers a 2-4x memory reduction with minimal accuracy impact
• Is the simplest quantization technique to implement
• Works well for models dominated by memory bandwidth constraints
• Trade-off: activation quantization happens on-the-fly, adding slight computational overhead compared to full integer quantization

A small, representative sample of unlabeled data used to determine optimal clipping ranges and scaling factors during post-training quantization. Critical properties:

• Typically 100-1000 samples from the target distribution
• Must reflect real-world inference data, not training data
• Used to observe activation value distributions and set quantization parameters
• Poor calibration data leads to clipping errors and accuracy degradation
• For medical devices, calibration data should mirror the specific patient population and sensor characteristics of the deployment environment

A model design paradigm that incorporates target deployment hardware constraints directly into the neural architecture search and training process. This goes beyond post-hoc quantization by:

• Simulating quantization noise during training with fake quantization nodes
• Optimizing for specific accelerator characteristics like NPU memory bandwidth
• Enabling quantization-aware training (QAT) that recovers accuracy lost in post-training methods
• Producing models that achieve near-floating-point accuracy at INT8 precision
• Essential for medical imaging models where even sub-1% accuracy drops are clinically unacceptable

A graph optimization strategy that combines multiple discrete neural network operations into a single computational kernel. This technique:

• Eliminates intermediate memory reads and writes between operations
• Reduces memory bandwidth bottlenecks on edge devices
• Commonly fuses convolution + batch normalization + activation functions
• Is performed automatically by inference engines like ONNX Runtime and TensorFlow Lite
• Synergizes with quantization: fused integer ops execute significantly faster than separate floating-point operations
• Critical for achieving real-time inference on battery-operated medical wearables

A complementary compression method that removes entire structural components of a neural network—such as channels, filters, or layers—rather than individual weights. Unlike unstructured pruning which creates sparse matrices, structured pruning:

• Produces a smaller, dense model readily accelerated by standard hardware
• Works synergistically with quantization for compound compression
• Reduces both compute and memory footprint simultaneously
• Enables predictable latency improvements on edge NPUs
• Typical approach: identify and remove low-magnitude channels, then fine-tune the pruned architecture
• Combined with INT8 quantization, can achieve 10x+ total model compression