Integer-Only Inference

Quantization maps continuous floating-point values (e.g., 32-bit floats) to a finite set of discrete integer levels. The core operation is:

• Scale (S): A floating-point multiplier that maps the integer range to the original float range.
• Zero-Point (Z): An integer offset that aligns the quantized range with the original, crucial for symmetric or asymmetric quantization. For a real value (r), its quantized integer (q) is calculated as: (q = \text{round}(r / S) + Z). Dequantization reconstructs an approximate value: (r' = S \cdot (q - Z)). This process introduces quantization error, which integer-only training aims to minimize.

The core challenge is replacing all floating-point operations with integer equivalents. Key transformations include:

• Integer Matrix Multiplication: Accumulate products in higher-bit integers (e.g., int32) to avoid overflow before rescaling.
• Fused Operations: Combine scaling, biasing, and activation functions into single integer steps.
• Rescaling with Fixed-Point: Multiplication by the scale factor (S) is implemented using fixed-point arithmetic—multiplying by an integer mantissa followed by a bit-shift, avoiding floating-point division. For example, a layer's output is computed as: (Y_{int32} = X_{int8} \cdot W_{int8}), then rescaled to int8 via: (Y_{int8} = \text{Requantize}(Y_{int32}, S_x, S_w, S_y)).

Non-linear functions like ReLU, Sigmoid, and Tanh must be approximated using lookup tables (LUTs) or piecewise integer polynomials.

• Lookup Tables (LUTs): Pre-compute integer outputs for all possible quantized inputs. Efficient but memory-intensive for high-precision inputs.
• Piecewise Linear Approximation: Break the function into segments approximated by simple integer operations: (y = \alpha \cdot x + \beta), where (\alpha) and (\beta) are pre-calculated integers.
• Hardware Intrinsics: Some NPUs provide dedicated integer units for common non-linearities. The choice of approximation directly trades off accuracy, latency, and memory footprint.

Two primary methodologies enable integer-only inference:

• Post-Training Quantization (PTQ): Converts a pre-trained FP32 model to integer using a small calibration dataset to determine scale/zero-point values. Faster but can incur accuracy loss, especially below 8-bit.
• Quantization-Aware Training (QAT): Simulates quantization during training by injecting fake quantization nodes. The model learns to compensate for quantization error, typically preserving higher accuracy. QAT is essential for complex models or aggressive quantization (e.g., to int4).

Deploying integer-only inference provides concrete system-level advantages:

• Reduced Memory Bandwidth: int8 weights occupy 4x less memory than FP32, cutting fetch energy.
• Lower Power Consumption: Integer ALUs are simpler, smaller, and more power-efficient than FPUs.
• Deterministic Latency: Integer ops have fixed cycle counts, unlike variable-latency FPUs.
• Broader Hardware Support: Runs on billions of microcontrollers lacking FPUs. Key Trade-off: The permanent introduction of quantization noise, which reduces model precision and must be managed via careful quantization strategy and training.

Specialized processors designed to accelerate integer tensor operations with extreme power efficiency.

• Arm Ethos-U55/U65: MicroNPUs for Cortex-M/A systems, supporting int8 and int4.
• Cadence Tensilica Vision P6: DSP with AI extensions for vision workloads.
• Google Edge TPU: ASIC for int8 inference at the network edge.
• These accelerators require a specific compiler toolchain (e.g., Ethos-U Vela compiler) to map and schedule integer graph operations onto the hardware.

The primary hardware for integer-only inference, characterized by severe constraints.

• Cortex-M Series (Arm): Dominant architecture; M4, M7, M33, M55 (with Helium vector extension).
• ESP32 (Tensilica Xtensa LX6): Popular for IoT, often coupled with a separate ultra-low-power coprocessor.
• RISC-V Cores: Emerging open-source alternatives with custom AI instruction set extensions.
• Key Constraints: Often < 1 MB SRAM, < 10 MB Flash, clock speeds < 500 MHz, and no operating system.

The multi-stage process of converting a floating-point model to executable integer code for a target device.

1. Model Quantization: Weights and activations are converted to int8 (or lower) via PTQ or QAT.
2. Graph Lowering & Optimization: The model graph is optimized (fusing ops, constant folding) for the target.
3. Code Generation: Low-level C/C++ or specialized bytecode is generated for the kernels.
4. Link & Deploy: The runtime and generated kernels are compiled with the device SDK into a single firmware binary.

A model compression technique where a neural network is trained with simulated low-precision (e.g., 8-bit integer) arithmetic. This allows the model to learn to compensate for the accuracy loss introduced by quantization, resulting in higher performance compared to Post-Training Quantization (PTQ) when deployed for integer-only inference.

• Key Mechanism: The forward pass simulates quantization (fake quantization), but the backward pass uses full-precision gradients.
• Result: Models are robust to the reduced numerical precision, making them ideal for TinyML deployment where every percentage of accuracy is critical.

A model compression technique that converts a pre-trained model's weights and activations from high-precision floating-point (e.g., 32-bit) to lower-precision integer (e.g., 8-bit) formats after training is complete. It is a faster, data-efficient method to enable integer-only inference but may incur a larger accuracy drop than QAT.

• Process: Requires a small calibration dataset to determine optimal scaling factors (quantization ranges).
• Primary Benefit: Dramatically reduces memory footprint and enables execution on hardware lacking Floating-Point Units (FPUs), which is standard for most Microcontroller Units (MCUs).

The layered software and hardware architecture required to develop, optimize, compile, and deploy machine learning models on ultra-low-power microcontrollers. The stack is what makes integer-only inference practically deployable.

• Typical Layers:
  • Model Design Frameworks (e.g., TensorFlow Lite for Microcontrollers)
  • Hardware-Aware Compilers (e.g., Apache TVM, MCU-optimized kernels)
  • Microcontroller Runtimes & Real-Time Operating Systems (RTOS)
• Function: Translates a quantized model into efficient, bare-metal code that respects the device's compute constraints and energy budget.

Performing mathematical computations using numerical formats with fewer bits than standard 32-bit floating-point (FP32). Integer-only inference is a subset focused on 8-bit or 16-bit integer (INT8/INT16) operations.

• Types: Includes integer (INT8), block floating-point, and reduced-bit floating-point (FP16, BF16).
• Impact: Reduces memory bandwidth, power consumption, and silicon area. Enables the use of simpler, more power-efficient processor cores common in resource-constrained devices.
• Hardware Support: Many modern Neural Processing Unit (NPU) accelerators are optimized for INT8/INT16 operations.

A lightweight software library deployed on a microcontroller or edge device that manages the local execution of the federated learning client protocol. For Federated Learning for TinyML, this runtime must support integer-only inference and potentially low-precision training.

• Responsibilities:
  • Loading the global model (often quantized).
  • Executing local training/inference using on-device data.
  • Managing secure communication and sparse updates.
  • Handling resource constraints and availability windows.
• Challenge: Must have an extremely small memory footprint to coexist with the application firmware.

The total amount of volatile (RAM) and non-volatile (Flash) memory consumed by a machine learning model's parameters, activations, and runtime buffers. This is the primary constraint that integer-only inference directly addresses.

• Quantization Effect: Converting from FP32 to INT8 reduces the model size by approximately 4x.
• Components: Includes model weights, activation memory for intermediate tensors, and code for the inference runtime.
• TinyML Target: Models must often fit within tens to hundreds of kilobytes of total memory on an MCU, making efficient footprint management non-negotiable.