Memory Footprint

The total number of learnable weights and biases defines the theoretical minimum storage requirement. Each parameter, typically stored as a 32-bit float, consumes 4 bytes. A ResNet-50 with ~25 million parameters requires ~100 MB just for its weights, before accounting for activations or the runtime framework. Architecture choice is the primary lever; a MobileNetV2 variant can achieve comparable diagnostic accuracy with a fraction of the parameters.

The bit-width of weights and activations directly scales memory usage. Moving from FP32 to INT8 post-training quantization reduces the model's weight footprint by 4x. FP16 or bfloat16 mixed precision halves memory relative to FP32 while preserving dynamic range for critical layers. Quantization-aware training can recover accuracy lost by aggressive INT8 conversion, making it a cornerstone of edge deployment.

Intermediate feature maps generated during a forward pass often dominate peak RAM usage, especially for high-resolution medical images. A 1024x1024 CT slice processed through a deep network can require hundreds of megabytes for activation storage. Activation checkpointing trades compute for memory by recomputing intermediate tensors during the backward pass, while gradient accumulation simulates larger batch sizes without storing full activation maps.

The inference engine itself consumes a baseline memory budget. TensorRT and ONNX Runtime maintain optimized execution graphs and CUDA contexts that add 50-200 MB of overhead. OpenVINO on Intel CPUs has a lighter footprint. Selecting a minimal runtime like TensorFlow Lite Micro or Apache TVM is critical for microcontroller-class devices where the OS and runtime must fit within a few hundred kilobytes of flash.

The spatial dimensions of the input tensor have a quadratic impact on activation memory. Downsampling a mammogram from 4K to 1K resolution before inference can reduce peak RAM by 16x. Similarly, a batch size of 1 is standard for edge deployment, as processing multiple scans simultaneously multiplies the activation footprint. Dynamic batching and image tiling strategies for gigapixel inference manage memory by processing sub-regions sequentially.

Unstructured pruning zeroes out individual weights, creating a sparse model that can be compressed via sparse matrix formats like CSR or CSC. Structured pruning removes entire channels or filters, yielding a smaller dense model that runs efficiently on standard hardware without specialized sparse kernels. A 90% sparse model can achieve a 10x reduction in storage, but realizing runtime speedup requires hardware support for sparse computation, such as NVIDIA's Ampere sparse tensor cores.

A compression technique that reduces the numerical precision of a neural network's weights and activations (e.g., from FP32 to INT8) to dramatically shrink memory footprint. Post-training quantization uses a small calibration dataset to convert a pre-trained model without retraining, while quantization-aware training simulates low-precision effects during fine-tuning for higher accuracy. This is often the first and most impactful step for deploying diagnostic models to edge hardware.

A model compression method where a compact student model is trained to replicate the output distribution of a larger, high-performance teacher model. The student learns not just the final classification but the teacher's softened probability scores, capturing inter-class relationships. This produces a smaller model with a significantly reduced memory footprint while preserving diagnostic accuracy, often outperforming training the small model from scratch.

A compression technique that removes entire structural components of a neural network—such as convolutional channels, filters, or entire layers—to create a smaller, dense model. Unlike unstructured pruning, which zeroes out individual weights, structured pruning produces a model that is immediately accelerated by standard hardware without specialized sparse computation libraries. This directly reduces both memory footprint and inference latency.

A model optimization paradigm that incorporates the specific constraints of the target deployment hardware—such as available SRAM, flash storage, and memory bandwidth—directly into the neural architecture search or training process. The optimizer receives feedback on memory footprint during training, enabling it to discover architectures that balance diagnostic accuracy with the strict memory budgets of embedded medical devices.