Activation Compression

This technique reduces the numerical precision of activation tensors from 32-bit floating-point (FP32) to lower bit-widths (e.g., INT8, INT4) during inference. It directly shrinks the memory footprint and bandwidth required to transfer activations between layers.

• Dynamic Quantization: Quantization scales are calculated on-the-fly per inference batch, adapting to input variability.
• Static Quantization: Scales are determined once using a calibration dataset, offering lower runtime overhead.
• Hardware Acceleration: Modern NPUs and GPUs have dedicated integer arithmetic units that execute quantized activations with significantly higher throughput and lower power consumption than floating-point operations.

This method identifies and zeros out less significant values within an activation tensor, creating a sparse activation map. Sparse tensors can be stored and processed more efficiently using specialized libraries and hardware that skip computations involving zeros.

• Sparsity Patterns: Can be unstructured (individual zeros) or structured (entire channels or blocks of zeros). Structured sparsity is often more amenable to hardware acceleration.
• Runtime Benefit: The computational savings are input-dependent; highly sparse activations lead to greater latency reduction.
• Use Case: Common in vision models processing natural images, where ReLU activations already induce significant sparsity that can be further amplified.

This is a memory-for-compute trade-off strategy to manage peak memory usage, crucial for devices with limited RAM. Instead of storing all intermediate activations for backpropagation during training, some are selectively discarded and later recomputed from a checkpoint.

• Gradient Checkpointing: A specific implementation where only activations at certain layer boundaries ('checkpoints') are stored. The intermediate ones are recomputed during the backward pass.
• Edge Inference Context: While primarily a training technique, similar principles apply to very deep models on edge devices, where storing all activations for a complex pipeline might exceed available memory, prompting selective offloading or recomputation strategies.

This advanced technique exploits the inherent low-dimensional structure in activation spaces. It uses matrix factorization (like Singular Value Decomposition) to approximate a large activation tensor with the product of smaller matrices.

• Mathematical Basis: Assumes activations have high correlation and can be represented in a lower-rank subspace without significant information loss.
• Application: More common in compressing weight matrices, but research explores applying it to activations in recurrent networks or attention mechanisms to reduce the cost of large intermediate states.
• Computational Overhead: The decomposition and reconstruction add overhead, so the net benefit is highly architecture-dependent.

Also known as early exiting or adaptive computation, this technique bypasses later layers in a network for 'easy' inputs where early-layer activations are already confident. It reduces the total number of activation tensors that need to be computed and stored.

• Internal Classifiers: Auxiliary classifiers are attached to intermediate layers. If confidence exceeds a threshold, inference halts, and that layer's activation is treated as the final output.
• Dynamic Computational Graph: The inference path becomes input-dependent, saving significant compute and memory on average.
• Edge Efficiency: Ideal for edge scenarios with highly variable input difficulty, such as a camera sensor that sees both simple and complex scenes.

This involves encoding activation tensors into proprietary, highly efficient formats designed for a specific accelerator's memory hierarchy and data paths. It is the final step where algorithmic compression meets hardware optimization.

• Sparse Encoding: Formats like CSR (Compressed Sparse Row) or CSC (Compressed Sparse Column) store only non-zero values and their indices.
• Blocked Formats: Data is rearranged into small, contiguous blocks (e.g., 4x4 or 8x8) optimal for a processor's cache lines and SIMD (Single Instruction, Multiple Data) units.
• Vendor Libraries: Frameworks like NVIDIA's TensorRT or Qualcomm's SNPE automatically apply optimal formatting during model compilation for their respective hardware, transparently compressing activations.

Quantization reduces the numerical precision of a model's weights and activations, typically from 32-bit floating-point to lower bit-width integers like INT8 or INT4. This directly decreases the model size and memory bandwidth requirements, accelerating inference on hardware that supports integer arithmetic.

• Post-Training Quantization (PTQ): Converts a pre-trained model using a calibration dataset.
• Quantization-Aware Training (QAT): Trains the model with simulated quantization for better accuracy retention.
• Impact: Can reduce model size by 4x (FP32 to INT8) and significantly cut memory bandwidth, which is critical for activation-heavy models.

Pruning removes redundant or less important parameters from a neural network to create a sparser, more efficient model. It reduces the parameter count and FLOPs.

• Structured Pruning: Removes entire structural components like filters or channels, producing a smaller, dense network that runs efficiently on standard hardware.
• Unstructured Pruning: Removes individual weights, creating an irregular sparse pattern that requires specialized runtimes or hardware (e.g., sparsity-supporting NPUs) for speedups.
• Hardware-Aware Pruning: Tailors the pruning strategy to the target hardware's architecture to maximize real-world latency improvements.

This technique trains a compact student model to replicate the behavior of a larger, more accurate teacher model. The student learns from the teacher's output distributions (soft labels), which often contain richer information than hard class labels.

• Primary Goal: To transfer the teacher's generalization capability and dark knowledge into a smaller, faster model suitable for edge deployment.
• Process: Unlike compression techniques applied to an existing model, distillation creates a new, efficient architecture from the ground up, guided by the teacher's knowledge.

This method approximates large weight matrices in a neural network as the product of two or more smaller matrices. It exploits the idea that weight matrices are often low-rank—meaning they contain redundant information.

• Mechanism: A weight matrix W of size m x n is factorized into matrices A (m x r) and B (r x n), where the rank r is much smaller than m and n.
• Benefit: Drastically reduces the number of parameters and multiplications. For example, a fully-connected layer's large matrix can be compressed, reducing both storage and compute.
• Common Use: Frequently applied to the large weight matrices in Transformer-based models.

NAS is an automated process for designing optimal neural network architectures under specific constraints like parameter count, latency, or FLOPs. It searches a vast space of possible operations and connections to discover efficient models inherently suited for edge devices.

• Relationship to Compression: Instead of compressing an existing large model, NAS directly constructs an efficient one. It is a form of design-time compression.
• Outcome: Architectures like EfficientNet and MobileNet are results of manual or automated NAS principles, achieving high accuracy with minimal computational cost.

This is a critical system-level metric that activation compression directly targets. The memory footprint is the total amount of memory required during inference, encompassing:

• Model Weights: The static parameters of the network (addressed by weight quantization and pruning).
• Activations: The dynamic intermediate layer outputs (addressed by activation compression).
• Runtime Buffers: Temporary memory used by the inference engine.

For edge devices with limited RAM (e.g., 1-2 GB), the activation memory for high-resolution inputs (e.g., 4K images) can be the dominant factor, making activation compression essential for feasible deployment.