CMSIS-NN

CMSIS-NN provides a library of hand-optimized C/C++ functions for fundamental neural network operations. These kernels are meticulously tuned for the Arm Cortex-M processor series, leveraging specific CPU features like the DSP extension and SIMD instructions (e.g., ARM's SIMD within a register - SWAR). This low-level optimization delivers significantly higher performance than naive implementations, enabling real-time inference on MHz-class processors with kilobytes of memory.

• Examples: arm_convolve_HWC_q7_fast() for INT8 convolutions, arm_fully_connected_q7() for dense layers.
• Benefit: Maximizes operations per clock cycle, directly translating to lower latency and power consumption.

The library is fundamentally designed for fixed-point and integer-only inference. It primarily uses INT8 (8-bit integer) and INT16 (16-bit integer) data types for weights and activations. This eliminates the need for a hardware Floating-Point Unit (FPU), which is absent or power-inefficient on most Cortex-M cores. Operations use symmetric quantization for weights and often asymmetric quantization for activations, with efficient handling of scaling factors and zero-points to maintain accuracy.

• Core Advantage: Enables high-performance neural network execution on the vast majority of cost-sensitive, power-constrained microcontrollers.

CMSIS-NN employs a static memory allocation strategy. All buffers for layer inputs, outputs, weights, and biases must be pre-allocated by the developer and passed into the kernel functions. This approach provides several critical benefits for embedded systems:

• Predictable RAM Usage: Peak memory footprint is known at compile-time, ensuring the model fits within the device's constraints.
• No Dynamic Allocation: Eliminates heap fragmentation and the runtime overhead of malloc/free, leading to deterministic execution timing.
• Facilitates In-Place Computation: Kernels can be designed to perform in-place computation, where output overwrites input buffers, further reducing peak RAM requirements.

As part of the Cortex Microcontroller Software Interface Standard (CMSIS), CMSIS-NN integrates seamlessly with the broader Arm ecosystem. It builds upon the CMSIS-DSP library for foundational math functions and follows consistent CMSIS coding conventions. This integration simplifies the development workflow for engineers already using CMSIS for peripheral drivers (CMSIS-Driver) or real-time operating system abstractions (CMSIS-RTOS).

• Ecosystem Benefit: Provides a unified, vendor-agnostic software layer that works across microcontrollers from all major silicon partners (ST, NXP, Microchip, etc.) implementing the Cortex-M architecture.

CMSIS-NN is rarely programmed directly. Instead, it serves as a critical backend optimization target for higher-level TinyML frameworks like TensorFlow Lite for Microcontrollers (TFLM) and Apache TVM. These frameworks' code generators can output calls to CMSIS-NN kernels when targeting a Cortex-M processor. This allows developers to design and train models in a high-level environment (e.g., TensorFlow) and automatically benefit from the low-level Arm optimizations during deployment, bridging the gap between ML research and production microcontroller deployment.

The library is optimized for the layer types most prevalent in microcontroller-scale models. This includes:

• Convolutions (standard and depthwise separable)
• Fully Connected (Dense) Layers
• Pooling (Average and Max)
• Activation Functions (ReLU, ReLU6, Sigmoid)
• Basic Element-Wise Operations

It does not implement every possible neural network operator, focusing instead on providing highly optimized implementations for the core set of operations that constitute efficient Embedded Neural Network Architectures like MobileNet and TinyML models. For complex or custom layers, developers may need to fall back to generic, unoptimized implementations.

Beyond mainstream frameworks, CMSIS-NN serves as a critical benchmarking baseline and integration target for research compilers and custom industrial toolchains.

• Research Compilers (e.g., those exploring neural architecture search for MCUs) often use CMSIS-NN latency models or integrate the library to evaluate the true performance of discovered networks.
• Custom Industrial Toolchains developed for specific product lines will often integrate CMSIS-NN directly, bypassing higher-level frameworks to achieve minimal overhead and full control over the static scheduling of kernels and memory.
• This demonstrates CMSIS-NN's role as a foundational building block for the broader TinyML ecosystem.

The numerical representation system at the heart of CMSIS-NN's efficiency. It uses Q-format notation (e.g., Q7, Q15) to represent fractional numbers as integers, avoiding the cost of hardware Floating-Point Units (FPUs).

• Qm.n Format: 'm' bits for the integer part (including sign), 'n' bits for the fractional part.
• CMSIS-NN Usage: Kernels are meticulously designed to perform operations in fixed-point, managing overflow and rounding. For example, a 16-bit multiply-accumulate might use a Q1.15 format for activations and a Q1.7 format for weights.
• Advantage: Enables high-performance inference on even the smallest Cortex-M0 cores that lack an FPU.

The model compression technique that enables the use of fixed-point arithmetic. CMSIS-NN kernels are designed to execute INT8 and sometimes INT16 quantized models.

• Symbiosis with CMSIS-NN: Quantization (e.g., via TFLM's converter) reduces model weights and activation tensors to 8-bit integers. CMSIS-NN's kernels are then the optimal runtime to compute with these integers.
• Per-Channel vs. Per-Tensor: CMSIS-NN supports efficient per-channel quantization for weights in convolutional layers, which uses a separate scale factor for each output channel for better accuracy.
• Pipeline: A model is quantized offline; the integer weights and quantization parameters (scales, zero-points) are stored in flash. CMSIS-NN loads them and performs all calculations in the integer domain.

A critical processor feature exploited by CMSIS-NN for parallelization at the register level. It allows one CPU instruction to operate on multiple data points simultaneously.

• Cortex-M Implementation: Through DSP extensions like the SIMD within a register (SWAR) pattern using 32-bit registers to hold four 8-bit values.
• CMSIS-NN Example: The arm_nn_mat_mult_kernel_q7_q15 kernel uses the SXTB16, SXTAB16, and SMLAD instructions to sign-extend and multiply-accumulate multiple 8-bit values in a single cycle.
• Performance Impact: This explicit use of SIMD in assembly/intrinsics is what gives CMSIS-NN its significant performance lead over generic C loops on the same hardware.

The memory management strategy essential for deterministic, real-time inference on MCUs. CMSIS-NN kernels are designed to work with pre-allocated buffers.

• Arena-Based Allocation: TFLM (when using CMSIS-NN) typically employs a single, statically allocated tensor arena—a large byte array in RAM.
• Kernel Design: CMSIS-NN functions accept pointers to input, output, weight, and scratch buffer memory. They do not perform dynamic allocation (malloc/free).
• Determinism: This eliminates heap fragmentation and allocation latency, guaranteeing consistent execution time and memory footprint, which is critical for safety-certifiable applications.