RAM Footprint

Activation tensors are the primary consumer of RAM during inference. They are the intermediate outputs produced by each layer of the neural network as data flows through the compute graph. The size of these tensors is dictated by the model's architecture—specifically the number of channels and spatial dimensions at each layer. Techniques like in-place computation and operator fusion are critical for reusing or eliminating these buffers to minimize peak memory usage.

Runtime buffers are temporary memory blocks allocated by the inference engine for scratch space during kernel execution. This workspace is used for operations like im2col transformations in convolutions or intermediate results in matrix multiplications. Its size can be substantial and is highly dependent on the implementation of optimized kernels (e.g., CMSIS-NN). Efficient static scheduling aims to overlap and reuse this workspace across operations to constrain its contribution to the overall footprint.

Memory management strategy is fundamental to RAM footprint predictability.

• Static Memory Allocation: All buffers (for weights, activations, workspace) are pre-allocated at compile-time. This eliminates runtime heap fragmentation and allocation overhead, providing deterministic memory usage essential for safety-critical systems.
• Dynamic Allocation: Memory is allocated from a heap at runtime. While flexible, it risks fragmentation and non-deterministic peak usage, making it unsuitable for most microcontroller deployments. Memory pooling is a hybrid approach that pre-allocates a large pool for runtime subdivision.

The neural network topology directly dictates the memory graph. Key architectural factors include:

• Layer Width & Sequence: Wider layers and sequential (non-branching) architectures create larger intermediate tensors.
• Operator Choices: Using depthwise separable convolutions over standard convolutions drastically reduces activation size. Avoiding operations that transpose large tensors (e.g., certain forms of attention) is also critical.
• Sparsity: While model pruning reduces the flash footprint of weights, exploiting sparsity for RAM reduction requires specialized runtime support to skip computations on zero activations.

The inference framework's compiler plays a decisive role in minimizing RAM. Key optimizations include:

• Operator Fusion: Merging consecutive ops (e.g., Conv + BatchNorm + ReLU) into a single kernel eliminates intermediate activation storage.
• In-Place Computation: Scheduling layers so the output overwrites a dead input buffer.
• Lifetime Analysis & Static Scheduling: The compiler analyzes tensor lifetimes across the entire compute graph to create a static memory plan that reuses memory across layers with non-overlapping lifetimes, minimizing the peak allocation.

Quantization primarily reduces the flash footprint (model size), but it also directly reduces RAM consumption. By converting activations from 32-bit floating-point (FP32) to 8-bit integers (INT8), the memory required to store each activation tensor is reduced by 4x. This applies to all intermediate buffers. INT8 inference is therefore a cornerstone of TinyML, as it simultaneously shrinks model size, accelerates computation via integer units, and slashes the peak RAM footprint.

Flash footprint refers to the total amount of non-volatile flash memory required to store a machine learning model's static assets. This includes:

• Model parameters (weights and biases)
• Model architecture metadata
• Constant tensors and lookup tables
• The inference engine runtime code itself

Unlike RAM, flash is used for persistent storage and is typically larger (hundreds of KB to MBs on MCUs). Optimization focuses on reducing model size via quantization and pruning.

Static memory allocation is a memory management strategy where all buffers for model weights, activations, and intermediate tensors are pre-allocated at compile-time. This approach:

• Eliminates runtime allocation overhead (e.g., malloc calls)
• Prevents memory fragmentation
• Guarantees deterministic execution, which is critical for real-time systems
• Allows the compiler to precisely calculate the total RAM footprint before deployment.

It is the standard method in frameworks like TensorFlow Lite Micro.

In-place computation is an optimization where the output of a neural network layer is written directly into the memory location of its input, reusing the buffer. This technique:

• Drastically reduces the peak RAM footprint by eliminating the need for separate input and output buffers for consecutive layers.
• Is applicable to operations where the input tensor is no longer needed after the layer executes (e.g., ReLU activation, certain pooling layers).
• Requires careful static scheduling by the compiler to ensure data integrity.

Operator fusion is a compiler optimization that combines multiple sequential neural network operations into a single, fused kernel. Common fusions include Convolution + BatchNorm + Activation. This process:

• Reduces the number of intermediate tensors that must be written to and read from RAM.
• Lowers the peak RAM footprint by avoiding the materialization of short-lived intermediate results.
• Decreases latency by minimizing memory bandwidth usage and kernel invocation overhead.
• Is a key optimization in compilers like Apache TVM and executed kernels in CMSIS-NN.

Memory pooling is a technique where a large, contiguous block of RAM is allocated once (often statically) and then subdivided into buffers for different inference tasks. This method:

• Improves allocation speed compared to dynamic allocation.
• Reduces memory fragmentation, ensuring more predictable RAM footprint over long runtimes.
• Can be used for variable-sized tensors by implementing a simple allocator within the pool.
• Is often used in conjunction with static scheduling to manage buffers for multiple models or complex data pipelines on a single MCU.

A compute graph is a directed acyclic graph (DAG) representation of a neural network, where nodes are operations and edges are tensors. Static scheduling uses this graph to:

• Determine a fixed, optimal execution order for all operations at compile-time.
• Perform lifetime analysis for every tensor, calculating the exact peak RAM footprint.
• Assign each tensor to a specific, reusable memory buffer (arena), minimizing total allocation.
• Eliminate all runtime scheduling decisions, resulting in minimal, deterministic overhead. This is the foundation for memory-efficient inference on microcontrollers.