CMSIS-NN

Provides a collection of hand-tuned, low-level neural network functions that map directly to Arm Cortex-M SIMD instructions. Key operations include:

• Convolution: Optimized 2D spatial filtering for feature extraction from spectrograms.
• Fully Connected: Efficient matrix-vector multiplication for classification heads.
• Pooling: Max and average pooling for dimensionality reduction.
• Activations: Fused ReLU, sigmoid, and tanh implementations to avoid redundant memory accesses. These kernels are specifically designed to exploit the Single Instruction Multiple Data (SIMD) capabilities of the Cortex-M4, M7, M33, and M55 processors, achieving significant speedups over naive C implementations.

CMSIS-NN is architected for INT8 and INT4 quantized models, aligning with the data types produced by frameworks like TensorFlow Lite Micro. The library operates on fixed-point arithmetic, avoiding the overhead of floating-point emulation on processors without an FPU.

• Weight and Activation Quantization: Kernels expect symmetric per-tensor or per-channel quantized parameters.
• Fused Operations: Combines convolution, batch normalization, and activation scaling into a single kernel call to reduce SRAM footprint and latency.
• Requantization Logic: Built-in routines efficiently handle the scaling and rounding required between layers to maintain numerical fidelity.

A primary design goal is minimizing peak SRAM footprint and flash usage, critical for microcontrollers with limited on-chip memory. CMSIS-NN achieves this through:

• In-Place Computation: Where possible, output buffers overwrite input buffers to conserve memory.
• Im2Col with Buffer Splitting: The image-to-column transformation for convolution is optimized to process data in chunks, avoiding large temporary matrices.
• Weight Sharing: Supports depthwise separable convolutions, which drastically reduce the parameter count and memory bandwidth requirements for mobile signal classifiers. These techniques enable complex models like neural receivers to fit within the sub-256KB SRAM constraints of typical Cortex-M devices.

The library defines a consistent API using C structures to describe layer parameters, enabling seamless integration with higher-level inference engines. Key structural elements include:

• cmsis_nn_dims: Describes tensor height, width, and channel count.
• cmsis_nn_context: Manages scratch buffer pointers required by kernels for intermediate computations.
• cmsis_nn_tile: Defines tiling parameters for breaking large operations into manageable blocks. This structured approach allows the Apache TVM compiler and TFLite Micro runtime to automatically target CMSIS-NN kernels, generating optimized code without manual intervention.

CMSIS-NN leverages processor-specific extensions to maximize TOPS/Watt efficiency:

• Cortex-M4/M7: Utilizes the SIMD instructions for dual 16-bit or quad 8-bit packed arithmetic.
• Cortex-M55: Integrates with the Arm Helium technology (M-Profile Vector Extension), providing up to a 5x performance uplift for machine learning workloads through 128-bit vector processing.
• Cortex-M85: Further extends Helium capabilities for higher throughput. The library automatically dispatches to the most optimized kernel variant based on the target architecture, ensuring peak performance for RF modulation classification or channel estimation tasks.

CMSIS-NN serves as the execution backend for models compressed using techniques like Post-Training Quantization (PTQ) and Weight Pruning. The library's efficiency is maximized when combined with:

• Batch Normalization Folding: Graph optimization that absorbs batch norm parameters into preceding convolution weights, which CMSIS-NN kernels execute as a single fused operation.
• Structured Pruning: N:M sparsity patterns that map efficiently to the vectorized load and store instructions of Cortex-M processors.
• Depthwise Separable Convolutions: A factorized layer type natively supported by CMSIS-NN to reduce the computational budget for on-device RF fingerprinting models.

A training methodology that simulates low-precision inference during the forward pass, enabling the model to learn parameters robust to quantization error. CMSIS-NN kernels are designed to execute these QAT-generated INT8 models with maximum efficiency.

• Simulates quantization noise during training
• Produces models that map directly to CMSIS-NN's integer-only functions
• Critical for maintaining accuracy on constrained microcontrollers

A precision reduction technique mapping 32-bit floating-point weights and activations to 8-bit integers. CMSIS-NN provides hand-optimized assembly routines specifically for INT8 convolution, depthwise, and fully-connected operations.

• Enables 2-4x speedup over FP32 on Cortex-M cores
• Uses symmetric per-tensor or per-channel quantization schemes
• Directly compatible with CMSIS-NN's arm_convolve_8x8 and related functions

A factorized convolution operation splitting standard spatial filtering into a depthwise step and a pointwise step. CMSIS-NN includes dedicated kernels for both operations, dramatically reducing the MAC count for mobile signal processing models.

• Depthwise kernel: arm_depthwise_conv_s8
• Pointwise kernel: arm_convolve_1x1_s8_fast
• Essential for deploying MobileNet-style architectures on Cortex-M

A graph optimization that mathematically absorbs batch normalization parameters into the preceding convolutional layer's weights and biases. This eliminates redundant runtime operations before executing on CMSIS-NN kernels.

• Reduces memory footprint by removing intermediate buffers
• Converts batch norm multiply-add into fused weight/bias terms
• Performed during model conversion, not at runtime

The peak amount of static random-access memory required to store intermediate activations, weights, and temporary buffers during inference. CMSIS-NN kernels are designed with minimal scratch memory requirements, often using in-place computation.

• Im2col buffer optimization reduces peak memory
• Chunked processing for large feature maps
• Critical for devices with < 512KB SRAM