Peak Memory Usage

The static parameters of a neural network, loaded into memory before inference. This is the largest, most predictable component.

• Quantization (e.g., INT8) directly reduces this footprint by storing weights in lower precision.
• For a 100KB model, weights consume ~100KB of read-only memory (Flash) and are copied to SRAM for execution unless executed directly from Flash (XiP).

The intermediate outputs (feature maps) from each network layer, stored temporarily during the forward pass. This is often the dominant variable cost in peak usage.

• Size depends on input dimensions and network architecture (e.g., large convolutional layers produce large activations).
• Operator fusion and in-place computation are key optimization techniques to reuse activation buffers and minimize this peak.

Memory allocated by the inference engine for workspace, scratch pads, and intermediate tensors for operations like convolutions or matrix multiplications.

• Includes memory for kernel im2col transformations, padding operations, and transposed layouts.
• This overhead is framework and kernel implementation dependent. A highly optimized runtime (e.g., TensorFlow Lite Micro, CMSIS-NN) minimizes these temporary buffers.

Defines when memory is reserved. Static allocation reserves all memory at compile-time, guaranteeing no runtime failures but potentially wasting space. Dynamic allocation (malloc/free) uses memory more efficiently but risks fragmentation and non-deterministic out-of-memory errors on MCUs.

• Most production TinyML deployments use static, arena-based allocators for deterministic behavior.

Advanced techniques to reduce the instantaneous peak by scheduling layer execution to share memory regions.

• Operator lifetime analysis: If the output of Layer A is only needed by Layer B, and A's input is no longer needed, they can share the same buffer.
• This requires compiler-level optimization (e.g., TVM, MLIR) to analyze the computational graph and generate a minimal, conflict-free memory plan.

Accurately measuring peak usage requires instrumentation. Common methods include:

• Linker Script Analysis: Inspecting the .bss and .data sections for static allocations.
• Runtime Heap Monitoring: Using custom allocators or Memory Protection Units (MPUs) to track high-water marks.
• Simulation/Emulation: Using tools like Renode or QEMU to profile memory access patterns before deploying to physical hardware.

The fixed amount of Read-Only Memory (ROM) or Flash consumed by the model's executable code, constant data, and most critically, its weights and biases. This is the baseline memory that must be permanently stored on the device. In TinyML, this is often the dominant memory component.

• Key Components: Model parameters (weights), constant literals, program code.
• Optimization Target: Reduced via quantization, pruning, and knowledge distillation.
• Contrast with Peak RAM: Static footprint is persistent; peak RAM is transient during inference.

The temporary Random-Access Memory (RAM) required to store the intermediate outputs (activations) of each layer in a neural network during a forward pass. This is a primary driver of peak memory usage.

• Calculation: Sum of the size of all intermediate tensors (height × width × channels × precision).
• Bottleneck: For deep networks, activation memory can far exceed the size of the weights.
• Optimization: Techniques include activation quantization, operator fusion to reuse buffers, and selecting network architectures with smaller activation maps.

A classification of a system's performance bottleneck. A workload is memory-bound when its speed is limited by the time to read/write data from memory, not by arithmetic speed. This is common in TinyML due to constrained memory bandwidth and large activation transfers.

• Memory-Bound Symptoms: High cache miss rates, low arithmetic intensity, underutilized compute units.
• Compute-Bound Symptoms: High MAC utilization, performance scales with clock speed.
• Implication: Optimizing for a memory-bound system focuses on data locality, kernel fusion, and reducing precision, not just raw FLOPs.

The maximum possible time a task (e.g., an inference) could take under all permissible operating conditions. For real-time TinyML systems, knowing WCET is essential for reliability.

• Relation to Memory: Memory hierarchy (cache hits/misses) and DRAM access latency are major contributors to timing variance.
• Analysis: Requires considering all possible code paths and data-dependent memory access patterns.
• Use Case: Critical for deterministic systems where missing a deadline is a failure.

An analytical performance model that visualizes the attainable performance of a computational kernel as a function of its operational intensity (Ops/Byte) and the hardware's peak compute and memory bandwidth.

• Axes: Performance (e.g., GOP/s) vs. Operational Intensity.
• The Roofline: A ceiling showing max performance limited by either compute (flat roof) or memory bandwidth (sloped roof).
• TinyML Application: Plots a model's layers to identify if they are compute-bound (near flat roof) or memory-bound (on the sloped roof), guiding optimization efforts.

The detailed measurement and analysis of resource consumption—time, memory, energy—for each individual layer or operator within a neural network. This granular data is key to diagnosing peak memory usage.

• Output: A breakdown showing which layers allocate the largest activation buffers or cause the most cache misses.
• Tools: Performed by specialized profiling tools (e.g., within TensorFlow Lite Micro, Arm Ethos-U NPU tools).
• Actionable Insight: Identifies specific layers to target for operator fusion, alternative implementation, or precision reduction.