Memory Footprint

The static, persistent data representing the learned knowledge of the network, stored as tensors. This is the baseline memory cost.

• Precision is critical: Storing weights in FP32 (4 bytes/param) vs. INT8 (1 byte/param) creates a 4x difference in memory.
• Example: A 7-billion parameter model requires ~26 GB in FP32 but only ~6.5 GB in INT8.
• This component is loaded once and remains resident in memory for the duration of inference.

The dynamic, temporary memory required to store intermediate outputs (activations) from each layer during a single forward pass.

• Scales with batch size and sequence length: Larger inputs produce larger intermediate tensors.
• A major bottleneck for large models: In transformer-based models, the Key-Value (KV) Cache for autoregressive generation can dominate memory usage, growing linearly with sequence length.
• Techniques like activation checkpointing (recomputing instead of storing) trade compute for memory reduction.

Memory allocated by the inference engine or framework for operational overhead.

• Includes space for kernel launches, intermediate computation results (like convolution workspaces), and framework-specific data structures.
• Hardware-dependent: NPUs and GPUs often require contiguous workspace memory for optimized kernel execution.
• This is often a hidden, non-negligible cost that must be profiled on the target hardware.

The memory consumed by the operating system, drivers, and the ML framework runtime itself.

• Includes the code footprint of the inference library (e.g., PyTorch Mobile, TFLite Runtime).
• Garbage collection and memory allocator fragmentation can introduce unpredictable overhead.
• Static vs. Dynamic linking of libraries can significantly affect this baseline footprint.

Memory reserved for the model's input data (e.g., an image tensor) and its final output predictions.

• Typically a small fraction of total footprint but must be accounted for in the total budget.
• Size is determined by the model's fixed input/output dimensions or the maximum supported size for dynamic shapes.
• For pipelines, this includes buffers between pre-processing, model execution, and post-processing stages.

Reducing memory footprint often involves trading off computational efficiency or accuracy.

• Quantization: Lowers weight and activation memory by reducing numerical precision, may impact accuracy.
• Pruning: Removes weights, reducing model size but can require sparse kernels for speed.
• Operator Fusion: Combines layers, eliminating intermediate activation buffers at the cost of fused kernel complexity.
• On-Demand Loading: Streaming weights from storage (e.g., SSD) instead of keeping all in RAM, drastically increases latency.

Quantization is a model compression technique that reduces the numerical precision of a neural network's weights and activations. By converting parameters from 32-bit floating-point (FP32) to lower bit-width formats like 8-bit integer (INT8) or 4-bit integer (INT4), it directly shrinks the model size and the memory needed to store parameters.

• Post-Training Quantization (PTQ): Converts a pre-trained model using calibration data, offering a quick path to a smaller footprint.
• Quantization-Aware Training (QAT): Trains the model with simulated quantization, often yielding higher accuracy for aggressive low-bit formats.
• Impact: Moving from FP32 to INT8 typically reduces the parameter storage footprint by 75%.

Pruning is a model compression technique that removes redundant or less important parameters (weights) or entire neurons from a neural network. This creates a sparse model architecture, reducing the number of parameters that must be stored in memory.

• Unstructured Pruning: Removes individual weights, creating an irregular sparse pattern that requires specialized hardware or libraries for speedup.
• Structured Pruning: Removes entire channels or filters, resulting in a smaller, denser model that runs efficiently on standard hardware.
• Goal: To eliminate parameters that contribute minimally to the model's output, thereby shrinking the parameter memory component of the total footprint.

Activation memory refers to the temporary storage required for the intermediate outputs (activations) of each layer during a forward pass. For deep networks or large batch sizes, this can be the dominant component of the peak memory footprint during inference, often exceeding the size of the model weights.

• Dependence on Sequence/Batch Size: Activation memory scales linearly with batch size and sequence length.
• Optimization Techniques: Include operator fusion (to avoid writing intermediates to memory), gradient checkpointing (in training), and algorithms like FlashAttention for Transformers, which use recomputation to trade compute for memory.
• Key Metric: Peak activation memory is a primary target for enabling inference of large models on memory-constrained devices.

The Key-Value (KV) Cache is a memory buffer used during autoregressive generation with Transformer models (like LLMs). It stores the computed keys and values for previous tokens in a sequence to avoid recomputing them for each new token, dramatically speeding up generation.

• Memory Growth: The KV cache size grows linearly with both batch size and sequence length, becoming a major memory bottleneck for serving.
• Optimizations: Techniques like PagedAttention (used in vLLM) manage the KV cache in non-contiguous blocks, allowing efficient memory sharing and reducing fragmentation.
• Footprint Impact: For large language models, managing the KV cache is often more critical for serving throughput and latency than the static model weights.

Operator fusion is a compiler-level optimization that combines multiple sequential neural network operations into a single, fused kernel. This reduces the memory footprint by eliminating the need to write and read intermediate tensors back to main memory (DRAM).

• Common Fusions: Convolution + Batch Normalization + Activation (ReLU) is a classic example.
• Benefits: Fewer kernel launches, reduced memory bandwidth pressure, and lower peak memory usage.
• Kernel Optimization: Complementary to fusion, this involves hand-tuning or auto-generating highly efficient low-level code (kernels) for specific hardware, maximizing data reuse within fast cache memory (cache locality) and minimizing costly DRAM accesses.

Model serialization is the process of converting a trained model's architecture, learned parameters, and execution graph into a standardized, persistent file format. This serialized file defines the deployable memory footprint.

• Formats: Common formats include ONNX (Open Neural Network Exchange), TorchScript, TensorFlow's SavedModel, and framework-specific .pt or .pth files.
• Optimization: Serialized models can be further optimized by inference runtimes (like ONNX Runtime or TensorRT), which apply graph-level optimizations (constant folding, dead code elimination) and quantize weights, resulting in a final runtime binary with a smaller effective memory footprint.
• Portability: A serialized model is the unit of transfer from training environments to edge deployment targets.