Memory Footprint

The model weights (or parameters) are the learned values of the neural network, stored persistently in non-volatile Flash or ROM memory. This is the largest component for most models. Its size is determined by:

• The number of parameters in the network.
• The numerical precision used (e.g., 32-bit float, 8-bit integer via quantization).
• For a 50,000-parameter model quantized to 8-bit integers, the weight footprint is approximately 50 KB.

Activation memory (or intermediate tensor memory) is the RAM required to store the outputs of each layer during a single inference pass. This is a peak memory requirement, not a persistent one. Key factors include:

• The size and number of feature maps or intermediate tensors.
• The network architecture's operator graph and the compiler's ability to perform in-place operations and operator fusion to reuse memory buffers.
• For image models, the first convolutional layer often creates the largest activation buffer.

The inference runtime and hardware-specific kernels are the executable code needed to perform tensor operations on the target MCU. This includes:

• The core inference engine (e.g., TensorFlow Lite for Microcontrollers, proprietary runtime).
• Optimized low-level kernels (e.g., CMSIS-NN libraries for Arm Cortex-M).
• Handlers for hardware accelerators (e.g., Ethos-U55 driver code).
• This footprint is typically fixed for a given runtime configuration and can range from 10-50 KB.

Input and output buffers are the RAM allocations for the model's raw input data (e.g., audio sample array, image pixel buffer) and the final output predictions (e.g., classification scores). Their size is defined by:

• The fixed dimensions of the model's input and output tensors.
• For a 96x96 RGB image input, the buffer size is 96 * 96 * 3 bytes = ~27.6 KB (for 8-bit pixels).
• These buffers are often separate from activation memory and are a mandatory, non-optimizable baseline RAM cost.

Memory allocation strategy profoundly impacts determinism and fragmentation.

• Static Allocation: All memory (weights, activations, I/O buffers) is allocated at compile-time. This eliminates heap usage, prevents fragmentation, and guarantees Worst-Case Execution Time (WCET), making it the standard for production TinyML.
• Dynamic Allocation: Memory is allocated from a heap at runtime (e.g., using malloc). This is flexible but risks heap fragmentation and non-deterministic out-of-memory failures, making it unsuitable for most long-lived, battery-powered embedded systems.

Memory alignment and efficient layout are low-level optimizations that reduce wasted space and improve access speed.

• Processors access memory most efficiently when data addresses are aligned to their word size (e.g., 4-byte alignment for 32-bit CPUs). Poor alignment creates padding (wasted bytes) and slower unaligned accesses.
• Tensor alignment in buffers and weight packing in Flash are critical optimizations performed by compilers like the TinyEngine in MCUNet to minimize the effective footprint.

A core compression technique that reduces the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). This directly shrinks the model's storage (Flash) and runtime (RAM) memory footprint and accelerates computation on hardware without native floating-point support.

• Post-Training Quantization (PTQ): Applied after a model is trained, often with minimal accuracy loss.
• Quantization-Aware Training (QAT): Simulates quantization during training for higher accuracy in the final low-precision model.

A memory management strategy where all buffers for model tensors, activations, and intermediate data are allocated at compile-time, not runtime. This is critical for TinyML on microcontrollers as it:

• Eliminates heap fragmentation and allocation overhead.
• Guarantees deterministic memory usage, enabling deployment on devices with kilobyte-level RAM.
• Allows the compiler to prove the entire inference workload fits within the device's physical memory constraints.

A compiler-level optimization that combines sequential neural network layers (operators) into a single, compound kernel. For example, fusing a Convolution, Batch Normalization, and ReLU activation into one operation. This optimization significantly reduces memory footprint by:

• Avoiding intermediate tensor writes to slow external memory.
• Keeping data in fast cache or registers between fused operations.
• Minimizing the peak RAM required during inference, which dictates the minimum RAM specification for the target hardware.

The maximum possible time a specific task, such as a single model inference, could take to complete on an edge device. WCET analysis is intrinsically linked to memory footprint because:

• Memory hierarchy access times (cache hit/miss, SRAM vs. Flash) dominate inference latency.
• A larger model footprint increases the likelihood of cache misses and slower memory accesses.
• For real-time TinyML systems, engineers must analyze WCET to guarantee deterministic performance, which requires understanding and controlling the memory access patterns.

A key system-level energy-efficiency metric measuring the number of successful inferences per joule of energy consumed. Memory footprint is a major driver of IPJ because:

• Memory accesses are power-expensive. Fetching data from Flash or DRAM consumes orders of magnitude more energy than a CPU arithmetic operation.
• A smaller, optimized footprint reduces the number and distance of these costly accesses.
• High IPJ is essential for battery-powered edge AI, directly linking model optimization to operational lifespan.

An automated process for designing neural networks that directly incorporates hardware constraints as objectives. For TinyML, the primary constraint is memory footprint. HW-NAS:

• Searches for model architectures that maximize accuracy while staying under strict RAM/Flash budgets.
• Profiles candidate models on target hardware (or simulators) to get accurate latency, energy, and memory usage metrics.
• Co-designs the model and the deployment runtime, leading to frameworks like MCUNet that achieve state-of-the-art accuracy on microcontrollers.