Integer-Only Inference

Integer arithmetic units (ALUs) are fundamentally more power-efficient than their floating-point (FPU) counterparts. By eliminating FPU usage, integer-only inference directly reduces the dynamic power consumption of the compute cores executing the model. This is critical for battery-powered devices, enabling longer operation and new use cases in always-on sensing. Power savings stem from simpler logic gates, lower transistor switching activity, and reduced data movement for lower-bitwidth integers.

Storing and moving 8-bit integers (INT8) uses 75% less memory than 32-bit floating-point (FP32) values. This compression directly translates to:

• Smaller model binaries for over-the-air updates and storage on memory-constrained devices.
• Reduced DRAM bandwidth, a common bottleneck that limits throughput and increases power. Lower bandwidth allows for the use of smaller, more efficient memory subsystems or enables higher inference throughput within the same power envelope.

Integer operations have deterministic, often single-cycle, latencies on most CPUs, DSPs, and NPUs. This contrasts with variable-latency floating-point pipelines. Benefits include:

• Lower and more predictable inference latency, essential for real-time applications like object detection for robotics or audio processing.
• Efficient use of fixed-function hardware like NPU tensor cores that are optimized for INT8/INT16 matrix multiplications, avoiding costly format conversions.

Virtually all computing devices, from high-end servers to the simplest microcontrollers, have native support for integer arithmetic. Integer-only models bypass the need for specialized floating-point hardware, enabling deployment on:

• Legacy and low-cost microcontrollers (MCUs) without FPUs.
• Digital Signal Processors (DSPs) optimized for fixed-point math.
• A wider range of mobile System-on-Chip (SoC) cores, ensuring consistent performance across device tiers. This maximizes the addressable market for an AI feature.

Removing floating-point operations eliminates a major source of numerical non-determinism across different hardware and software backends. Floating-point results can vary slightly between CPU architectures, compilers, or math libraries due to rounding modes and fused operations. Integer arithmetic is bit-exact across platforms, guaranteeing reproducible results—a key requirement for debugging, validation, and regulatory compliance in sectors like automotive or medical devices.

Integer-only inference is the foundational runtime for more aggressive compression techniques. Once a model operates in the integer domain, it becomes feasible to apply:

• Lower bit-width quantization (e.g., INT4, INT2) with specialized integer kernels.
• Sparse integer computation, where pruning creates zeros that skip integer multiply-accumulate (MAC) operations.
• Binary/ternary neural networks, which use purely bitwise logic. These advanced methods build upon the integer data path to push the boundaries of efficiency.

A training process where quantization error is simulated during the forward pass, allowing the model to learn parameters robust to the precision loss of integer deployment. This is often a prerequisite for high-accuracy integer-only inference.

• Key Mechanism: Inserts fake quantization nodes into the computational graph.
• Benefit: Produces models that maintain significantly higher accuracy after post-training quantization compared to standard training.

A numerical representation system where numbers have a fixed number of fractional bits, enabling efficient integer-based computation. It is the mathematical foundation for executing quantized neural networks on hardware lacking floating-point units (FPUs).

• Core Concept: Uses integer registers to represent numbers scaled by a constant power of two.
• Hardware Advantage: Eliminates the need for complex, power-hungry FPUs, making it ideal for microcontrollers and dedicated NPUs.

Two fundamental schemes for mapping floating-point values to integers. The choice directly impacts the efficiency and accuracy of integer-only inference kernels.

• Symmetric Quantization: Range is symmetric around zero. Simplifies hardware by eliminating the zero-point, often used for weights.
• Asymmetric Quantization: Uses separate scale and zero-point parameters. More accurately represents data not centered around zero, typically used for activations.

A critical compiler optimization that combines multiple sequential operations (e.g., Conv2D, BatchNorm, ReLU) into a single, fused kernel. This is essential for performance in integer-only inference.

• Primary Benefit: Dramatically reduces memory bandwidth pressure by eliminating intermediate tensor writes to DRAM.
• Impact on Integer Math: Allows for constant folding and bias pre-computation within the integer domain, reducing overall operations.

Hand-tuned, low-level software routines that implement neural network operations (like matrix multiplication) to exploit the unique features of a target processor (e.g., NPU vector units, GPU tensor cores).

• Purpose: Maximize throughput and minimize power consumption for integer operations.
• Examples: Kernels optimized for ARM NEON instructions on mobile CPUs or proprietary DSP libraries for Qualcomm Hexagon.