Flash Footprint

This is the largest component, containing the learned parameters of the neural network. After quantization (e.g., to INT8), these are stored as integer arrays. The size is determined by:

• Number of parameters: The total count of weights and biases.
• Bit-width: Precision after quantization (e.g., 8-bit vs. 4-bit).
• Sparsity: If model pruning is applied, a sparse format may store only non-zero values and indices, potentially reducing size.

This defines the compute graph—the sequence of layers and operations. It includes:

• Layer metadata: Type (Conv2D, FullyConnected), dimensions, stride, padding.
• Static hyperparameters: Non-learned values like epsilon for normalization layers.
• Quantization parameters: Scaling factors and zero-points for each quantized tensor, essential for dequantization during or after inference.

The software library that executes the model. For microcontrollers, this is often a statically linked library like TensorFlow Lite Micro (TFLM) or CMSIS-NN. Its size depends on:

• Supported operators: Only kernels for used ops are included via compiler linking.
• Optimization level: Hand-optimized assembly kernels (e.g., for SIMD instructions) may be larger than reference C code.
• Memory manager: A simple static memory allocator adds minimal overhead versus a dynamic one.

Algorithms that prepare input data and interpret output. This is often custom C/C++ code and includes:

• Sensor front-end: Digital signal processing for audio, FFT, image scaling, normalization.
• Output decoding: Translating neural network output tensors into actionable results (e.g., parsing bounding boxes, applying a softmax, keyword spotting).
• Fixed-point math libraries: For operations not handled by the inference engine.

Persistent memory allocations defined at compile-time. While activations (intermediate results) reside in RAM, some frameworks may statically allocate:

• Input/Output buffers: Fixed locations for sensor data and final results.
• Look-up tables: Pre-computed tables for complex functions (e.g., sigmoid, tanh) used in fixed-point arithmetic to avoid costly runtime calculations.

The final footprint is influenced by compiler and framework choices:

• Operator Fusion: Combining ops (Conv + BatchNorm + ReLU) into a single kernel reduces code size.
• Link-Time Optimization (LTO): Dead code elimination strips unused runtime functions.
• Flash vs. RAM Trade-off: Techniques like weight streaming can store compressed weights in flash and decompress to RAM, trading flash size for runtime RAM and compute.

RAM footprint refers to the peak amount of volatile working memory (Random Access Memory) required during the execution of a machine learning model to store activations, intermediate tensors, and runtime buffers on a microcontroller. Unlike flash memory for storage, RAM is used for live computation.

• Key Drivers: Layer output sizes, batch size, and any temporary buffers allocated by the inference engine.
• Critical Constraint: Microcontrollers often have only 10s to 100s of KB of RAM, making this a primary bottleneck for model architecture selection.
• Optimization Techniques: In-place computation, operator fusion, and static memory allocation are used to minimize peak RAM usage.

Quantization is a foundational model compression technique that reduces the numerical precision of a neural network's weights and activations. It directly reduces flash footprint and accelerates inference.

• Standard Precision: Models are typically trained in 32-bit floating-point (FP32).
• Quantized Precision: Deployed using 8-bit integers (INT8) or even 4-bit integers, reducing model size by 4x or more.
• Two Main Approaches: Post-training quantization (PTQ) converts a trained model, while quantization-aware training (QAT) simulates quantization during training for higher accuracy.
• Impact on Flash: A 5 MB FP32 model becomes a ~1.25 MB INT8 model, making MCU deployment feasible.

Static memory allocation is a memory management strategy where all buffers for model weights, activations, and intermediate tensors are pre-allocated at compile-time. This is essential for deterministic, real-time execution on microcontrollers.

• Eliminates Overhead: Removes the latency and code size of dynamic memory allocators (malloc/free).
• Prevents Fragmentation: Guarantees memory availability for the lifetime of the program, crucial for long-running embedded applications.
• Enables Optimization: Allows the compiler to precisely analyze and optimize memory layout, often intertwining with static scheduling of the compute graph.

Kernel optimization refers to the manual or automated low-level tuning of fundamental neural network operation implementations for a specific microcontroller's CPU architecture. It maximizes performance and minimizes cycle count.

• Targets Core Ops: Focuses on convolution, fully-connected layers, and activation functions.
• Key Techniques:
  • Loop unrolling to reduce branch overhead.
  • Loop tiling to improve cache locality.
  • SIMD instructions to process multiple data points per cycle.
• Frameworks: Libraries like CMSIS-NN (for Arm Cortex-M) provide highly optimized kernels that are hand-written in assembly or intrinsic C.

Model pruning is a compression technique that removes redundant or less important parameters (weights) from a neural network. It reduces both flash footprint (for storing weights) and computation.

• Creates Sparsity: Pruned weights are set to zero, creating a sparse weight matrix.
• Structured vs. Unstructured:
  • Unstructured pruning removes individual weights; requires sparse kernels for acceleration.
  • Structured pruning removes entire channels/filters; results in a smaller, dense model easier to deploy.
• Typical Workflow: Train a large model → prune → fine-tune to recover accuracy → deploy a smaller model.