Memory Footprint

The static, persistent memory required to store the learned parameters of the model. This is the baseline size of the saved model file (e.g., .pt, .tflite).

• Primary determinant of storage footprint.
• Directly targeted by compression techniques like quantization (e.g., FP32 to INT8), pruning, and weight clustering.
• Example: A 50-million parameter model in FP32 occupies ~200 MB. Quantized to INT8, it reduces to ~50 MB.

The dynamic, temporary memory required to store intermediate layer outputs during a single forward pass (inference).

• Scales with batch size and input resolution (e.g., high-definition images create large activations).
• Can become the dominant memory consumer for deep networks or large inputs, even if weights are small.
• Optimized via activation quantization, activation pruning, and careful layer-by-layer memory scheduling.

Memory allocated by the inference engine (e.g., TensorFlow Lite, ONNX Runtime) for operational overhead.

• Includes memory for kernel execution workspace, temporary tensors for operations like convolutions, and I/O buffers.
• Size is highly dependent on the hardware backend (CPU, GPU, NPU) and the kernel implementations.
• A key focus for compiler-level optimizations to minimize transient allocations.

The memory used to store the model's computational graph structure, operator definitions, tensor shapes, and other metadata.

• Generally a small, fixed overhead compared to weights and activations.
• Includes information required by the runtime to execute the model correctly.
• Format efficiency varies between frameworks (e.g., FlatBuffers in TFLite minimize this overhead).

A critical distinction for resource planning on edge devices.

• Static Memory: The persistent storage for the model weights and graph. This is the size of the deployed file.
• Peak Memory: The maximum RAM consumed at any point during inference. This is the sum of weights (if loaded), activations, and runtime buffers at the most memory-intensive layer.
• System design must guarantee available RAM > Peak Memory to avoid out-of-memory crashes.

The final memory consumption is dictated by the target hardware's memory hierarchy.

• SRAM vs. DRAM: Moving weights/activations from slow DRAM to fast SRAM/cache is critical for performance but SRAM is severely limited.
• Unified Memory: Architectures with CPU/GPU/NPU sharing memory simplify management but increase contention.
• Compressed Weight Streaming: Some NPUs can execute directly from compressed weights in DRAM, reducing SRAM pressure.
• Techniques like hardware-aware pruning are designed to align sparsity patterns with the hardware's efficient execution capabilities.

Quantization is a model compression technique that reduces the numerical precision of a neural network's weights and activations. By converting values from 32-bit floating-point (FP32) to lower bit-width formats like 16-bit (FP16/BF16) or 8-bit integers (INT8), it directly shrinks the model size and the memory required for activations during inference.

• Post-Training Quantization (PTQ): Converts a pre-trained model using a calibration dataset.
• Quantization-Aware Training (QAT): Trains the model with simulated quantization for higher accuracy.
• Impact: Reduces memory footprint by 4x (FP32 to INT8) and accelerates computation on integer-optimized hardware.

Pruning is a model compression technique that removes redundant or less important parameters from a neural network to reduce its parameter count and computational graph size. The resulting sparse model has a smaller memory footprint for storing weights.

• Structured Pruning: Removes entire structural units (e.g., filters, channels) for efficient execution on standard hardware.
• Unstructured Pruning: Removes individual weights, creating an irregular sparse pattern that requires specialized libraries or hardware (e.g., sparse tensor cores) for speedup.
• Goal: Eliminate weights that contribute minimally to the output, often measured by magnitude or gradient-based importance scores.

Activation compression refers to techniques that reduce the memory cost of intermediate layer outputs (activations) during the forward pass. For models processing high-resolution inputs (e.g., images, long sequences), the memory for storing activations can exceed the memory for the model weights, becoming a critical bottleneck.

• Methods: Include activation quantization, sparsification, and recomputation (where activations are recalculated during backpropagation instead of stored).
• Edge Relevance: Directly reduces peak RAM usage during inference, which is essential for devices with limited memory bandwidth and capacity.

Model sparsification is the overarching process of inducing sparsity (a high percentage of zero values) in a neural network's parameters, often via pruning. A sparse tensor is the data structure used to efficiently represent these pruned models.

• Storage: Only non-zero values and their indices are stored, dramatically reducing the memory footprint compared to a dense representation.
• Computation: Enables skip-over-zero operations, reducing FLOPs. However, realizing speedups requires software frameworks (e.g., TensorFlow Lite with sparsity) or hardware with sparse compute support.
• Trade-off: High sparsity ratios can lead to irregular memory access patterns, potentially hurting performance on some hardware.

Efficient model architectures are neural networks designed from the ground up for low memory and computational cost, making them inherently suitable for edge deployment. They represent a design-time approach to minimizing memory footprint.

• Examples: MobileNet (uses depthwise separable convolutions), EfficientNet (uses neural architecture search and compound scaling).
• Principle: Employ architectural innovations that maintain representational power while drastically reducing the number of parameters and activation sizes compared to standard models like ResNet.
• Use Case: Often serve as the base models for further compression via quantization or pruning.

Hardware-aware optimization tailors model compression strategies to the specific memory hierarchy, cache sizes, and compute units of the target edge processor. The goal is to maximize real-world inference speed and efficiency, not just theoretical parameter reduction.

• Considerations: Alignment of data structures for efficient loading, leveraging specialized instructions (e.g., INT8 vector units), and choosing a pruning structure (e.g., channel pruning) that maps cleanly to the hardware's parallel execution model.
• Toolchain: Involves using hardware-specific compilers and SDKs (e.g., NVIDIA TensorRT, Qualcomm SNPE, ARM Compute Library) that perform final graph optimizations and memory planning for the target device.