Static Quantization

TensorFlow Lite is Google's primary framework for deploying models on mobile and edge devices. Its static quantization pipeline is central to TinyML.

• TFLite Converter: The tf.lite.TFLiteConverter is used to apply post-training quantization (PTQ), converting FP32 models to INT8 using a representative calibration dataset.
• Integer-Only Deployment: TFLite supports full integer inference kernels, eliminating floating-point operations entirely for compatible ops.
• TFLite Micro: A subset of TFLite designed for microcontrollers, featuring a bare-metal interpreter and highly optimized kernels for Arm Cortex-M series processors. It is the de facto standard for deploying quantized models on MCUs.
• Key Feature: Supports per-channel quantization for convolutional weights, which often yields better accuracy than per-tensor quantization.

PyTorch provides a comprehensive quantization toolkit for research and production, with a growing focus on edge deployment.

• torch.ao.quantization: The PyTorch quantization API supports both Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). Static quantization is performed via torch.quantization.prepare and torch.quantization.convert.
• ExecuTorch: PyTorch's next-generation runtime for on-device inference, designed for portability across diverse edge hardware. It natively supports models quantized via PyTorch's APIs.
• Mobile Interpreter: The PyTorch Mobile interpreter executes the quantized model file (.ptl or .pte) efficiently on Android and iOS.
• Key Advantage: Seamless workflow from research model in PyTorch to a quantized, deployable asset, maintaining dynamic graph flexibility during the quantization process.

Arm's microcontroller and microNPU architectures are the dominant hardware targets for quantized TinyML workloads.

• Cortex-M Processors: The Cortex-M55 and Cortex-M85 introduce the Arm Helium technology (M-Profile Vector Extension), providing significant acceleration for INT8 inference compared to earlier M-series cores.
• Arm Ethos-U55/U65 NPUs: These are microNPUs designed to sit alongside a Cortex-M CPU, providing dedicated hardware for quantized neural network computations. They deliver orders of magnitude higher performance and efficiency for INT8 models.
• CMSIS-NN: Arm's optimized library of neural network kernels for Cortex-M processors. It provides hand-tuned assembly/C functions for operations like convolution and pooling on INT8 and INT16 data, forming the computational backbone for TFLite Micro on these chips.

The Open Neural Network Exchange (ONNX) ecosystem provides a framework-agnostic path for static quantization and high-performance inference.

• ONNX Quantization Tools: Python tools (onnxruntime.quantization) provide calibration-based static quantization for models from any framework exported to ONNX format.
• ONNX Runtime: A cross-platform inference engine that includes highly optimized execution providers (EPs) for quantized models. EPs like CPU (with MLAS kernels) and TensorRT leverage integer arithmetic.
• Hardware Vendor Integration: Many hardware accelerators (e.g., Intel Neural Compute Stick, Qualcomm Hexagon) provide their own ONNX Runtime EPs, accepting quantized ONNX models as a standard input format.
• Key Role: Serves as a vital interoperability layer, allowing a model quantized in PyTorch or TensorFlow to be deployed across a wide array of server and edge hardware.

Popular IoT SoCs that include hardware features beneficial for running statically quantized models.

• Espressif ESP32-S3: Features an Ultra-Low-Power (ULP) core and vector instructions for INT8 operations, making it a common target for audio-based TinyML applications (e.g., keyword spotting).
• Cadence Tensilica Vision & Fusion DSPs: These are licensable DSP cores found in many IoT and vision processors. They include instruction set extensions (HIFI series) specifically designed for efficient 8-bit and 16-bit fixed-point arithmetic, crucial for running quantized CNNs.
• Vendor SDKs: Companies like Espressif provide customized versions of TFLite Micro and optimized kernels that leverage their specific hardware extensions.

Qualcomm's Snapdragon platforms for mobile and IoT use dedicated hardware and software for quantized inference.

• Hexagon DSP: A digital signal processor within Snapdragon SoCs featuring Hexagon Vector eXtensions (HVX). HVX is a wide SIMD architecture extremely efficient for INT8 tensor operations.
• Qualcomm Neural Processing SDK (QNN SDK): The software stack that compiles models (from ONNX, TFLite) for execution on Hexagon DSPs, Adreno GPUs, and Kryo CPUs. It performs advanced graph optimizations and leverages Hexagon's integer math capabilities.
• AI Engine Direct: A newer, unified software interface for accessing all compute cores (CPU, GPU, DSP) with support for quantized models.
• Target Use Case: Enables high-performance, power-efficient execution of statically quantized models on always-on mobile and IoT applications.

A training-time compression technique where quantization error is simulated during the forward and backward passes. Unlike static PTQ, the model learns to adapt its weights to the lower-precision format, typically yielding higher accuracy for aggressive quantization schemes (e.g., INT4).

• Process: Fake quantization nodes are inserted into the training graph. These nodes quantize and dequantize tensors, mimicking inference-time behavior while allowing gradients to flow.
• Use Case: Essential when post-training quantization causes unacceptable accuracy drops, especially for models with non-linear activations or highly sensitive layers.