Glossary
On-Device Model Compression
Neural Network Quantization
Terms related to reducing the numerical precision of model weights and activations to shrink memory footprint and accelerate integer compute. Target: ML Engineers, Embedded Systems Developers.
Quantization-Aware Training (QAT)
Quantization-Aware Training (QAT) is a model compression technique that simulates quantization effects during the training process to improve the final accuracy of a low-precision model.
Post-Training Quantization (PTQ)
Post-Training Quantization (PTQ) is a model compression technique that reduces the numerical precision of a pre-trained model's weights and activations without requiring retraining.
Integer Quantization
Integer quantization is a model compression technique that constrains a neural network's weights and activations to integer values, enabling efficient execution on hardware with native integer arithmetic units.
Quantization Scale and Zero-Point
The quantization scale and zero-point are parameters that define the affine transformation mapping floating-point values to a quantized integer range, enabling both symmetric and asymmetric quantization schemes.
Symmetric Quantization
Symmetric quantization is a scheme where the quantized range is symmetric around zero, simplifying computation by setting the zero-point parameter to zero.
Asymmetric Quantization
Asymmetric quantization is a scheme where the quantized range is mapped to the actual minimum and maximum values of the tensor, often providing a tighter fit and less quantization error for asymmetric data distributions.
Per-Channel Quantization
Per-channel quantization is a granularity scheme where a unique scale and zero-point are calculated for each output channel of a weight tensor, typically yielding higher accuracy than per-tensor quantization.
Calibration (Quantization)
Calibration in quantization is the process of analyzing a representative dataset to estimate the optimal range (min/max) or distribution statistics for determining quantization parameters like scale and zero-point.
Quantization Error
Quantization error is the numerical distortion or loss of information introduced when mapping a continuous range of floating-point values to a finite set of discrete integer levels.
Straight-Through Estimator (STE)
The Straight-Through Estimator (STE) is a method used during Quantization-Aware Training that approximates the gradient of the non-differentiable quantization function, allowing error backpropagation.
Quantization Bit-Width
Quantization bit-width refers to the number of bits used to represent each quantized value, with common targets being 8-bit (INT8), 4-bit (INT4), or 16-bit (FP16) precision.
Mixed-Precision Quantization
Mixed-precision quantization is a technique that assigns different numerical precisions (bit-widths) to different layers, channels, or operators within a model to optimize the trade-off between accuracy and efficiency.
Dequantization
Dequantization is the process of converting quantized integer values back into floating-point numbers using the scale and zero-point parameters, often required for operations between tensors of different precisions.
Quantization Clipping (Saturation)
Quantization clipping, or saturation, is the process of constraining values that fall outside the representable quantized range to the nearest representable limit (min or max) to prevent overflow during arithmetic operations.
Dynamic Quantization
Dynamic quantization is a Post-Training Quantization method where the quantization parameters (scale/zero-point) for activations are calculated at runtime based on the actual data observed during inference.
Static Quantization
Static quantization is a Post-Training Quantization method where the quantization parameters for all activations are predetermined during a calibration step and fixed for all inference runs.
Quantization-Aware Fine-Tuning
Quantization-aware fine-tuning is a process of lightly retraining a pre-trained model with simulated quantization to recover accuracy lost after applying Post-Training Quantization.
Uniform Quantization
Uniform quantization is a scheme where the quantization levels are evenly spaced across the represented range, leading to a constant step size between adjacent quantized values.
Non-Uniform Quantization
Non-uniform quantization is a scheme where quantization levels are not evenly spaced, allowing a higher density of levels in regions where the value distribution is more concentrated to reduce error.
Quantization Grid
The quantization grid is the finite set of discrete integer values that represent the quantized output levels within a specified range and bit-width.
Weight Quantization
Weight quantization is the process of reducing the numerical precision of a neural network's stored weight parameters, directly reducing the model's memory footprint.
Activation Quantization
Activation quantization is the process of reducing the numerical precision of a neural network's intermediate layer outputs (activations), reducing memory bandwidth and enabling fully quantized integer operations.
Rounding Mode (Quantization)
The rounding mode in quantization defines the rule for mapping a floating-point value to the nearest discrete quantized level, with common modes being round-to-nearest and stochastic rounding.
Quantization Folding
Quantization folding is a graph optimization technique that merges or 'folds' linear operations like batch normalization into preceding layers before quantization, simplifying the quantized graph and improving accuracy.
Bias Correction (Quantization)
Bias correction is a post-quantization method that adjusts the bias terms of layers to compensate for the systematic error introduced by quantizing the weights, reducing overall quantization error.
TFLite Quantization
TFLite Quantization refers to the suite of quantization tools and runtime support provided by TensorFlow Lite for deploying efficient integer models on mobile and edge devices.
Binary Quantization
Binary quantization is an extreme form of quantization where weights or activations are constrained to just two values, typically +1 and -1, enabling highly efficient bitwise operations.
Quantization for Neural Processing Units (NPUs)
Quantization for Neural Processing Units (NPUs) involves tailoring quantization schemes to exploit the specific low-precision integer arithmetic units and data paths of dedicated AI accelerators.
Model Pruning Techniques
Terms related to the systematic removal of redundant or non-critical parameters from a neural network to create a sparse architecture. Target: ML Engineers, Performance Architects.
Model Pruning
Model pruning is a neural network compression technique that systematically removes redundant or non-critical parameters to create a sparse, more efficient architecture.
Structured Pruning
Structured pruning is a model compression technique that removes entire structural components, such as neurons, channels, or filters, to maintain hardware-friendly dense matrix operations.
Unstructured Pruning
Unstructured pruning is a model compression technique that removes individual weights anywhere in a network, resulting in irregular sparsity patterns that require specialized hardware or libraries for efficient inference.
Magnitude-Based Pruning
Magnitude-based pruning is a heuristic pruning technique that removes network parameters with the smallest absolute values, under the assumption they contribute less to the model's output.
Iterative Magnitude Pruning (IMP)
Iterative Magnitude Pruning (IMP) is a pruning algorithm that alternates between training a model and pruning a fraction of the smallest-magnitude weights over multiple cycles to recover accuracy.
Lottery Ticket Hypothesis
The Lottery Ticket Hypothesis is a theory stating that within a dense, randomly-initialized neural network, there exists a sparse subnetwork (a 'winning ticket') that, when trained in isolation, can match the performance of the original network.
Sparse Networks
Sparse networks are neural architectures where a significant proportion of the connections (weights) are zero, a state typically achieved through pruning to reduce computational cost and memory footprint.
Pruning Rate
The pruning rate is the percentage or fraction of a model's parameters targeted for removal during a pruning operation, directly controlling the final sparsity level.
Pruning Schedule
A pruning schedule is a policy that defines the progression of the pruning rate over time, such as when and how many parameters to prune during iterative training.
Channel Pruning
Channel pruning is a form of structured pruning that removes entire output channels from convolutional layers, directly reducing the width of feature maps and the computational cost of subsequent layers.
Filter Pruning
Filter pruning is a form of structured pruning that removes entire convolutional filters, reducing the number of output channels and the computational load of the layer.
Global Pruning
Global pruning is a strategy that ranks and removes parameters across the entire network based on a single, unified importance criterion, rather than pruning each layer independently.
Local Pruning
Local pruning is a strategy that ranks and removes parameters independently within each layer or group of a network, often using a per-layer threshold or sparsity target.
N:M Sparsity
N:M sparsity is a structured sparsity pattern where, in every block of M consecutive weights, at most N are non-zero, enabling efficient execution on modern sparse tensor cores in GPUs.
Post-Training Pruning
Post-training pruning is a compression workflow where a pre-trained model is pruned without any subsequent retraining, often leading to significant accuracy loss unless the model exhibits inherent redundancy.
Pruning-Aware Training
Pruning-aware training is a model development paradigm where pruning constraints or regularizers are applied during the initial training process to encourage the emergence of sparsity-friendly representations.
Sparse Training
Sparse training is a technique where a neural network is initialized and trained with a fixed sparse connectivity pattern from the start, avoiding the dense pre-training phase of traditional pruning.
Gradual Magnitude Pruning (GMP)
Gradual Magnitude Pruning (GMP) is a pruning schedule that slowly increases sparsity from an initial rate (often 0%) to a final target over many training steps, allowing the network to adapt and maintain accuracy.
Movement Pruning
Movement pruning is a gradient-based pruning technique that removes weights based on how much their value changes (moves) during fine-tuning, rather than their final static magnitude.
Optimal Brain Damage (OBD)
Optimal Brain Damage (OBD) is an early second-order pruning algorithm that uses a diagonal approximation of the Hessian matrix to estimate the saliency of parameters for removal.
Pruning Mask
A pruning mask is a binary matrix of the same shape as a weight tensor that indicates which parameters are active (1) and which have been pruned (0), defining the sparsity pattern.
Pruning Granularity
Pruning granularity refers to the structural unit at which parameters are removed, ranging from fine-grained (individual weights) to coarse-grained (entire layers or blocks).
Sparsity Pattern
A sparsity pattern defines the specific locations of zero-valued elements within a weight tensor or network, which can be unstructured, structured (e.g., N:M), or block-wise.
Rewinding
Rewinding is a technique in iterative pruning where the network's weights are reset to an earlier checkpoint in training (but not the initial random values) after a pruning step, before retraining continues.
Sparsity-Accuracy Tradeoff
The sparsity-accuracy tradeoff describes the inverse relationship between the level of sparsity (compression) induced in a model and its resulting predictive performance on a given task.
Hardware-Aware Pruning
Hardware-aware pruning is a model compression approach that selects sparsity patterns and granularities specifically to maximize efficiency on a target hardware accelerator's execution engine and memory hierarchy.
Pruning for Inference
Pruning for inference is the application of pruning techniques with the primary goal of reducing the computational cost, latency, and memory usage of a model during its deployment phase.
Neuron Pruning
Neuron pruning is a form of structured pruning that removes entire neurons (units) from fully connected layers, effectively reducing the layer's width.
Pruning Criterion
A pruning criterion is the metric or heuristic used to score and rank parameters for removal, such as weight magnitude, gradient information, or activation statistics.
Knowledge Distillation
Terms related to training a smaller, more efficient student model to mimic the behavior of a larger, more powerful teacher model. Target: ML Researchers, Engineers building compact models.
Knowledge Distillation
Knowledge distillation is a model compression technique where a smaller, more efficient student model is trained to mimic the behavior and outputs of a larger, more powerful teacher model.
Teacher-Student Framework
The teacher-student framework is the foundational architecture for knowledge distillation, consisting of a pre-trained teacher model that transfers its knowledge to a trainable student model.
Logits Distillation
Logits distillation is a form of knowledge distillation where the student model is trained to match the raw, pre-softmax output logits of the teacher model.
Soft Targets
Soft targets are the probability distributions output by a teacher model's softmax layer, which contain richer, inter-class similarity information (dark knowledge) compared to hard, one-hot labels.
Temperature Scaling
Temperature scaling is a technique used in knowledge distillation where a temperature parameter (T) is applied to the softmax function to soften the teacher model's output probability distribution, making it easier for the student to learn.
Dark Knowledge
Dark knowledge refers to the implicit, inter-class relationships and similarities captured within a trained neural network's softened output probabilities, which are transferred to a student model during distillation.
Attention Transfer
Attention transfer is a feature-based distillation method where the student model is trained to replicate the spatial attention maps generated by intermediate layers of the teacher model.
Feature Distillation
Feature distillation is a knowledge transfer method where the student model is trained to match the intermediate feature representations or activations from specific layers of the teacher model.
Online Distillation
Online distillation is a training paradigm where the teacher and student models are co-trained simultaneously from scratch, rather than using a static, pre-trained teacher.
Self-Distillation
Self-distillation is a knowledge distillation variant where a model serves as both the teacher and the student, often by distilling knowledge from deeper layers to shallower layers within the same architecture.
Multi-Teacher Distillation
Multi-teacher distillation is a technique where a single student model learns from an ensemble of multiple teacher models, aggregating knowledge from diverse sources or architectures.
Distillation Loss
Distillation loss is the specialized objective function, typically based on Kullback-Leibler divergence or mean squared error, used to measure and minimize the difference between the teacher and student model's outputs or features.
Kullback-Leibler Divergence (KL Divergence)
Kullback-Leibler divergence is a statistical measure of how one probability distribution diverges from a second, reference distribution, and is the most common loss function for logit-based knowledge distillation.
Data-Free Distillation
Data-free distillation is a technique to perform knowledge transfer without access to the original training dataset, often by using synthetic data generated by the teacher model or via adversarial methods.
Hint Learning
Hint learning is an early feature distillation technique where the student is guided by intermediate representations (hints) from the teacher's network, as introduced in the FitNets architecture.
Relational Knowledge Distillation
Relational knowledge distillation is a method where the student learns to mimic the relationships or correlations between different data samples or feature representations as modeled by the teacher.
Born-Again Networks
Born-again networks are a specific self-distillation technique where a student model with the same architecture as the teacher is trained to outperform its teacher through iterative knowledge distillation cycles.
Mutual Learning
Mutual learning is a collaborative training paradigm where multiple peer student models teach each other simultaneously, improving collectively without a static, superior teacher model.
Policy Distillation
Policy distillation is the application of knowledge distillation techniques to reinforcement learning, where a compact student policy network is trained to replicate the behavior of a larger or ensemble teacher policy.
DistilBERT
DistilBERT is a prominent, general-purpose distilled version of the BERT language model, developed by Hugging Face, that retains 97% of its language understanding capabilities while being 40% smaller and 60% faster.
TinyBERT
TinyBERT is a task-specific distilled version of BERT that employs transformer distillation at both the pre-training and task-specific fine-tuning stages to achieve high compression rates for on-device deployment.
DeiT (Data-efficient Image Transformer)
DeiT is a vision transformer model that achieves competitive performance without requiring massive datasets by leveraging a native distillation token and a teacher-student training strategy during pre-training.
Attention Distillation
Attention distillation is a technique for compressing transformer models where the student is trained to replicate the self-attention patterns or matrices of the teacher model's layers.
Federated Knowledge Distillation
Federated knowledge distillation is a privacy-preserving distributed learning technique where clients train local models and distill their knowledge into a central server model without sharing raw data.
Quantization-Aware Distillation
Quantization-aware distillation is a joint optimization technique where knowledge distillation is performed during training while simulating the effects of quantization, preparing the student model for efficient integer deployment.
Pruning-Aware Distillation
Pruning-aware distillation is a combined compression strategy where the student model is trained with distillation objectives while also learning to operate effectively with a sparse, pruned architecture.
Contrastive Distillation
Contrastive distillation is a knowledge transfer method that uses contrastive learning objectives to help the student model learn a representation space that aligns with the teacher's, often improving feature discrimination.
Cross-Modal Distillation
Cross-modal distillation is a technique where knowledge is transferred from a teacher model trained on one data modality (e.g., image) to a student model designed for a different modality (e.g., text).
Low-Rank Factorization
Terms related to decomposing large weight matrices into smaller, more efficient factors to reduce parameter count. Target: ML Researchers, Algorithm Optimizers.
Low-Rank Factorization
Low-rank factorization is a model compression technique that decomposes a large weight matrix into the product of two or more smaller matrices, reducing the total number of parameters and computational cost.
Singular Value Decomposition (SVD)
Singular Value Decomposition (SVD) is a fundamental matrix factorization method that decomposes any matrix into three constituent matrices, revealing its singular values and vectors, which is foundational for low-rank approximation.
Truncated SVD
Truncated SVD is a low-rank approximation technique that retains only the top-k largest singular values and corresponding vectors from a full SVD to create a compressed representation of the original matrix.
Principal Component Analysis (PCA)
Principal Component Analysis (PCA) is a dimensionality reduction and feature extraction technique that uses eigenvalue decomposition of a covariance matrix to project data onto its principal components, which are orthogonal directions of maximum variance.
Canonical Polyadic Decomposition (CPD)
Canonical Polyadic Decomposition (CPD), also known as PARAFAC or CANDECOMP, is a tensor factorization method that expresses a tensor as a sum of rank-one tensors, defined by factor matrices for each mode.
Tucker Decomposition
Tucker decomposition is a higher-order generalization of SVD for tensors, factorizing a tensor into a core tensor multiplied by a factor matrix along each mode, enabling multilinear dimensionality reduction.
Tensor-Train Decomposition
Tensor-train decomposition is a tensor factorization format that represents a high-dimensional tensor as a sequence of interconnected low-dimensional core tensors, mitigating the curse of dimensionality for storage and computation.
Non-Negative Matrix Factorization (NMF)
Non-Negative Matrix Factorization (NMF) is a matrix factorization technique that constrains all factor matrices to have non-negative entries, often used for parts-based representation and feature learning in data like images or text.
Independent Component Analysis (ICA)
Independent Component Analysis (ICA) is a computational method for separating a multivariate signal into additive, statistically independent non-Gaussian source components, commonly used for blind source separation.
Eckart–Young Theorem
The Eckart–Young theorem states that the optimal rank-k approximation to a matrix in the Frobenius or spectral norm is given by the truncated singular value decomposition (SVD) retaining the k largest singular values.
Nuclear Norm
The nuclear norm, or trace norm, is the sum of a matrix's singular values, serving as a convex surrogate for matrix rank and a key regularizer in low-rank matrix completion and robust PCA problems.
Low-Rank Matrix Completion
Low-rank matrix completion is the problem of recovering the missing entries of a partially observed matrix under the assumption that the complete matrix has low rank, often solved using nuclear norm minimization.
Robust PCA
Robust Principal Component Analysis (Robust PCA) is a variant of PCA that decomposes a matrix into a low-rank component and a sparse component, making it resilient to outliers and corruptions in the data.
Alternating Least Squares (ALS)
Alternating Least Squares (ALS) is an optimization algorithm commonly used for matrix and tensor factorization that alternates between fixing subsets of variables and solving a least squares problem for the others.
Randomized SVD
Randomized SVD is a computationally efficient algorithm for approximating the truncated singular value decomposition of large matrices using random projection and subspace iteration techniques.
Nyström Method
The Nyström method is a technique for approximating large kernel matrices and their eigendecompositions by using a subset of columns, enabling scalable kernel methods and low-rank approximations.
CUR Decomposition
CUR decomposition is a matrix approximation method that expresses a matrix as a product of a subset of its columns (C), a subset of its rows (R), and a linking matrix (U), providing an interpretable, column/row-based low-rank factorization.
Krylov Subspace Methods
Krylov subspace methods are a class of iterative algorithms for solving large-scale eigenvalue problems and linear systems, which construct approximations from the sequence of matrix-vector products, forming the basis for methods like Lanczos and Arnoldi iteration.
Lanczos Algorithm
The Lanczos algorithm is an iterative Krylov subspace method for approximating a few extreme eigenvalues and eigenvectors of a large, sparse, symmetric matrix by tridiagonalizing it.
Power Iteration
Power iteration is a simple iterative algorithm used to find the dominant eigenvector (the eigenvector corresponding to the eigenvalue with the largest magnitude) of a diagonalizable matrix.
Rayleigh Quotient
The Rayleigh quotient is a scalar value associated with a given vector and a Hermitian matrix, providing an estimate for an eigenvalue and serving as the foundation for the Rayleigh quotient iteration algorithm.
Schatten p-Norm
The Schatten p-norm of a matrix is the p-norm of its vector of singular values, generalizing the nuclear norm (p=1), Frobenius norm (p=2), and spectral norm (p=∞), used in various matrix regularization schemes.
Proximal Gradient Method
The proximal gradient method is an optimization algorithm for minimizing an objective function that is the sum of a differentiable term and a potentially non-differentiable term (like the nuclear norm), using a proximal operator step.
Iterative Hard Thresholding (IHT)
Iterative Hard Thresholding (IHT) is a greedy algorithm for sparse approximation and low-rank matrix recovery that iteratively performs a gradient step followed by hard thresholding to enforce a sparsity or rank constraint.
Higher-Order SVD (HOSVD)
Higher-Order SVD (HOSVD) is a multilinear generalization of the matrix SVD for tensors, decomposing a tensor into a core tensor and orthogonal factor matrices for each mode, forming the basis for Tucker decomposition.
Tensor Rank
Tensor rank, specifically the CP rank, is the minimum number of rank-one tensors required to exactly represent a given tensor as their sum, a central but computationally challenging concept in tensor factorization.
Random Projection
Random projection is a dimensionality reduction technique that projects data onto a random lower-dimensional subspace, approximately preserving pairwise distances (Johnson-Lindenstrauss lemma) and enabling fast low-rank approximations.
Proper Orthogonal Decomposition (POD)
Proper Orthogonal Decomposition (POD), equivalent to PCA in certain contexts, is a method for extracting a low-dimensional basis from high-dimensional snapshot data, commonly used in fluid dynamics and model order reduction.
Singular Value Thresholding
Singular value thresholding is an operation central to nuclear norm minimization, where a matrix is soft-thresholded by applying a shrinkage function to its singular values, used in algorithms for matrix completion and robust PCA.
Hardware-Aware Compression
Terms related to model optimization techniques that are co-designed with or specifically target the characteristics of underlying silicon (e.g., NPUs, mobile SoCs). Target: Embedded AI Engineers, Silicon Architects.
Hardware-Aware Compression
Hardware-aware compression is a model optimization strategy that tailors neural network compression techniques, such as quantization and pruning, to the specific architectural features and performance characteristics of a target hardware accelerator, like an NPU or mobile SoC.
Quantization-Aware Training (QAT)
Quantization-aware training is a process where a neural network is trained or fine-tuned with simulated quantization operations, allowing the model to learn parameters that are robust to the precision loss incurred during subsequent integer-only deployment.
Post-Training Quantization (PTQ)
Post-training quantization is a compression technique that converts a pre-trained floating-point model to a lower-precision format (e.g., INT8) using calibration data, without requiring retraining, to reduce model size and accelerate inference.
Per-Channel Quantization
Per-channel quantization is a method where scaling factors for quantization are calculated independently for each output channel of a weight tensor, providing finer-grained control and typically lower error compared to per-tensor quantization.
Dynamic Range Calibration
Dynamic range calibration is the process of analyzing the statistical distribution (e.g., min/max values) of a model's activations over a representative dataset to determine optimal quantization parameters like scale and zero-point.
Integer-Only Inference
Integer-only inference is an execution mode where all operations of a neural network, including activations, are performed using integer arithmetic, eliminating the need for floating-point units and reducing power consumption on edge devices.
Mixed-Precision Quantization
Mixed-precision quantization is a strategy that assigns different numerical bit-widths (e.g., 4-bit, 8-bit, 16-bit) to different layers or tensors within a model to optimize the trade-off between accuracy, latency, and model size.
Weight Clipping
Weight clipping is a pre-quantization technique that constrains the range of weight values to a predefined limit, reducing outlier-induced quantization error and improving the effectiveness of post-training quantization.
Symmetric Quantization
Symmetric quantization is a scheme where the quantization range is symmetric around zero, simplifying the dequantization formula by eliminating the need for a separate zero-point and often enabling more efficient hardware implementation.
Asymmetric Quantization
Asymmetric quantization is a scheme that uses separate scale and zero-point parameters to map the floating-point range to the integer range, allowing for a more precise representation of data that is not centered around zero.
Fixed-Point Arithmetic
Fixed-point arithmetic is a numerical representation system where numbers have a fixed number of digits after the radix point, enabling efficient integer-based computation for quantized neural networks on hardware without floating-point units.
Binary Neural Networks (BNN)
Binary neural networks are an extreme form of quantization where weights and activations are constrained to +1 or -1, enabling highly efficient inference using primarily bitwise XNOR and popcount operations.
Hardware-Specific Kernels
Hardware-specific kernels are low-level, optimized software routines written to exploit the unique architectural features of a particular processor, such as tensor cores on a GPU or vector units on an NPU, for maximum performance.
Tensor Core Mapping
Tensor core mapping is the process of structuring computational graph operations, particularly matrix multiplications and convolutions, to efficiently utilize the dedicated, high-throughput tensor core units found in modern AI accelerators.
SIMD Optimization
SIMD optimization is the technique of restructuring code to leverage Single Instruction, Multiple Data (SIMD) processor instructions, which perform the same operation on multiple data points simultaneously, to accelerate vector and matrix operations.
Memory-Bound Optimization
Memory-bound optimization focuses on improving the performance of computational workloads that are limited by the speed of memory access, through techniques like data layout transformation, prefetching, and cache blocking.
Compute-Bound Optimization
Compute-bound optimization focuses on improving the performance of workloads limited by the processor's arithmetic capabilities, often by increasing operational intensity, leveraging specialized compute units, and maximizing parallelism.
Operator Fusion
Operator fusion is a compiler optimization that combines multiple sequential operations (e.g., convolution, batch normalization, activation) into a single kernel, reducing intermediate memory writes and kernel launch overhead.
Graph Compilation
Graph compilation is the process of transforming a high-level neural network computational graph into a highly optimized, executable program tailored for a specific target hardware backend.
Kernel Auto-Tuning
Kernel auto-tuning is an automated process that searches a space of possible kernel implementations and parameter configurations (e.g., tile sizes, unroll factors) to find the optimal version for a given hardware platform and workload size.
Hardware Intrinsics
Hardware intrinsics are compiler functions that provide direct, low-level access to specific processor instructions (e.g., NEON, AVX-512), allowing developers to write highly optimized code for critical loops without using assembly language.
TensorRT Optimization
TensorRT optimization refers to the suite of techniques applied by NVIDIA's TensorRT SDK, including layer fusion, precision calibration, and kernel auto-selection, to maximize inference performance of deep learning models on NVIDIA GPUs.
CoreML Optimization
CoreML optimization refers to the model conversion and acceleration techniques applied by Apple's CoreML framework to deploy and efficiently execute machine learning models on Apple silicon (CPU, GPU, and Neural Engine).
NNAPI Delegation
NNAPI delegation is a mechanism in Android where the Neural Networks API runtime can offload supported operations of a model to dedicated hardware accelerators (NPUs, DSPs, GPUs) for faster and more power-efficient execution.
TFLite for Microcontrollers
TFLite for Microcontrollers is a lightweight version of TensorFlow Lite designed to run machine learning models on microcontrollers and other devices with only kilobytes of memory, featuring a minimal core runtime and offline code generation.
TVM Compilation Stack
The TVM compilation stack is an open-source compiler framework that takes models from various frontends (PyTorch, TensorFlow) and compiles them for optimal execution across a wide range of hardware backends (CPUs, GPUs, accelerators).
MLIR Dialects
MLIR dialects are domain-specific sets of operations and types within the MLIR compiler infrastructure, such as the 'TOSA' dialect for tensor operations or custom hardware dialects, used to represent and transform computational graphs for different targets.
Hardware Abstraction Layer (HAL)
A hardware abstraction layer in AI compilation is a software interface that decouples high-level model operations from low-level hardware-specific kernel implementations, allowing a single model representation to target multiple accelerators.
Vendor SDKs
Vendor SDKs are software development kits provided by silicon manufacturers (e.g., Qualcomm SNPE, Intel OpenVINO) that contain proprietary tools, libraries, and compilers to optimize and deploy AI models on their specific hardware platforms.
On-Device Calibration
On-device calibration is the process of running a representative dataset through a model on the target hardware to gather activation statistics, which are then used to calculate final quantization parameters for optimal accuracy on that specific device.
Sparse Model Inference
Terms related to the execution kernels, hardware support, and runtime optimizations required to efficiently run pruned, sparse neural networks. Target: Systems Engineers, Kernel Developers.
Sparse Tensor Representation
A data structure for efficiently storing and operating on tensors where the majority of elements are zero, typically using formats like CSR, CSC, or COO to encode only non-zero values and their indices.
CSR Format (Compressed Sparse Row)
A sparse matrix storage format that compresses row indices, storing arrays for column indices of non-zero values, the non-zero values themselves, and compressed row pointers.
COO Format (Coordinate Format)
A sparse tensor storage format that explicitly stores a list of (row, column, value) tuples for each non-zero element, offering simplicity but potentially higher memory overhead for operations.
Structured Pruning
A model compression technique that removes entire structural components of a neural network, such as channels, filters, or layers, to produce hardware-friendly, regular sparsity patterns.
Unstructured Pruning
A model compression technique that removes individual weights from a neural network based on a saliency criterion, resulting in irregular, fine-grained sparsity that requires specialized hardware or kernels for efficient execution.
N:M Sparsity Pattern
A structured sparsity constraint where in every block of M consecutive weights, only N are allowed to be non-zero, enabling efficient execution on modern GPU tensor cores with native sparse compute support.
Weight Sparsity
The property of a neural network where a significant fraction of its weight parameters are exactly zero, typically induced through pruning to reduce model size and computational cost.
Activation Sparsity
The property where the output activations of a neural network layer contain a high proportion of zero values, often induced by activation functions like ReLU, which can be exploited to skip computations.
Sparse Matrix Multiplication (SpMM)
A fundamental computational kernel that multiplies a sparse matrix by a dense matrix, requiring specialized algorithms to skip operations involving zero elements and efficiently gather-scatter data.
Sparse Convolution
A convolution operation where either the input feature map, the kernel weights, or both are sparse, allowing for the skipping of multiplications with zero values to accelerate computation.
Sparse Attention
An approximation of the standard attention mechanism in transformer models that computes pairwise interactions only for a selected subset of query-key pairs, drastically reducing the quadratic computational and memory complexity.
SpMM Kernel
A highly optimized software routine, often implemented for GPUs or specialized accelerators, that performs sparse matrix-dense matrix multiplication by leveraging zero-skipping and efficient memory access patterns.
Gather-Scatter Operations
Fundamental parallel computing primitives where 'gather' collects data from non-contiguous memory addresses into a contiguous vector, and 'scatter' writes a contiguous vector to non-contiguous addresses, essential for sparse computation.
Sparse Tensor Core
A specialized hardware unit within modern GPUs (e.g., NVIDIA Ampere/Ada/Hopper architectures) designed to accelerate sparse matrix operations by leveraging structured 2:4 sparsity patterns to effectively double theoretical compute throughput.
Zero-Skipping
The foundational optimization in sparse inference where computations involving zero-valued operands (weights or activations) are identified and omitted, reducing the number of floating-point operations (FLOPs) required.
Sparse Data Layout
The specific in-memory arrangement of non-zero values and their corresponding indices (e.g., CSR, CSC, Blocked) which critically determines the performance of sparse kernels by influencing memory access patterns and cache efficiency.
Bitmask Encoding
A method for representing sparsity patterns using a compact sequence of bits, where each bit indicates whether a corresponding weight or activation is zero (0) or non-zero (1), enabling efficient pruning masks and runtime checks.
Pruning Mask
A binary matrix or tensor that has the same shape as a model's weight tensor, with values indicating which parameters are active (1) and which have been pruned (0), used to freeze sparsity during fine-tuning or guide runtime execution.
Sparse Fine-Tuning
The process of retraining a pruned neural network after sparsity has been induced, where the pruning mask is typically fixed to maintain the sparse structure while updating the remaining non-zero weights to recover accuracy.
Magnitude-Based Pruning
A prevalent unstructured pruning algorithm that removes weights with the smallest absolute magnitudes, operating under the hypothesis that these contribute least to the model's output.
Sparse Inference Engine
A software runtime or framework component (e.g., in TensorFlow Lite, PyTorch, or proprietary SDKs) specifically designed to load and execute sparse neural network models with optimized kernels for target hardware.
Sparse CUDA Kernels
Custom CUDA functions written to execute sparse linear algebra operations on NVIDIA GPUs, optimizing thread parallelism, memory coalescing, and warp-level operations to handle irregular data access.
Load Imbalance
A major performance challenge in parallel sparse computation where different processing threads or cores are assigned vastly different amounts of work due to the irregular distribution of non-zero elements.
Sparse FLOPs
A count of the actual floating-point operations required to execute a sparse model, which is lower than the theoretical dense FLOPs count due to zero-skipping, though actual speedup depends on memory bandwidth and kernel overhead.
Sparse Efficiency Gap
The observed performance difference between the theoretical speedup predicted by the reduction in FLOPs and the actual speedup achieved on hardware, caused by overheads like metadata processing and irregular memory access.
Sparse Kernel Overhead
The additional computational cost incurred during sparse execution from operations such as index decoding, pointer chasing, conditional branching, and gather-scatter instructions, which can diminish the benefits of zero-skipping.
Sparse Model Profiling
The practice of measuring and analyzing the performance characteristics of a sparse model, including layer-wise sparsity, memory bandwidth utilization, cache misses, and kernel execution time to identify bottlenecks.
Sparse Operator Fusion
A compiler optimization that combines multiple consecutive sparse operations (e.g., a sparse linear layer followed by a ReLU activation) into a single kernel to reduce intermediate memory reads/writes and kernel launch overhead.
Sparse Hardware Mapping
The process undertaken by a compiler to map the abstract sparse computational graph of a model onto the specific execution units, memory hierarchy, and ISA extensions of a target accelerator (e.g., NPU, GPU).
Sparse Quantization
The combined application of pruning (to induce sparsity) and quantization (to reduce numerical precision) to a neural network, yielding multiplicative reductions in model size and potential compute acceleration.
Compute Graph Optimization
Terms related to transforming and fusing a model's computational graph for optimal execution on target hardware. Target: Compiler Engineers, Inference Framework Developers.
Operator Fusion
Operator fusion, also known as kernel fusion or layer fusion, is a compiler optimization technique that combines multiple sequential operations in a neural network's computational graph into a single, more efficient kernel to reduce memory bandwidth usage and kernel launch overhead.
Constant Folding
Constant folding is a compile-time optimization that evaluates and replaces expressions consisting entirely of compile-time constants with their precomputed result, eliminating runtime computation and simplifying the execution graph.
Dead Code Elimination
Dead code elimination is a compiler optimization pass that identifies and removes operations in a computational graph whose outputs do not affect the final result of the program, thereby reducing execution time and memory usage.
Common Subexpression Elimination
Common subexpression elimination (CSE) is a compiler optimization that identifies and eliminates redundant calculations of identical expressions within a computational graph, caching and reusing the computed result to improve efficiency.
Graph Lowering
Graph lowering is the process of transforming a high-level, hardware-agnostic intermediate representation (IR) of a neural network into a lower-level, target-specific IR or machine code that can be efficiently executed by a particular hardware backend.
Intermediate Representation (IR)
An intermediate representation (IR) is an abstract, platform-independent data structure or code used within a compiler to represent a program, such as a neural network's computational graph, between its source form and its final executable form.
Loop Tiling
Loop tiling, also known as loop blocking, is a loop nest optimization that partitions loop iterations into smaller blocks or tiles to improve data locality and cache utilization, which is critical for performance on memory-bound operations like matrix multiplication.
Loop Unrolling
Loop unrolling is a loop optimization technique that reduces loop control overhead by duplicating the body of a loop multiple times, decreasing the number of iterations and increasing opportunities for instruction-level parallelism and compiler optimization.
Vectorization
Vectorization is a compiler optimization that converts scalar operations, which process a single data element at a time, into vector operations that process multiple data elements simultaneously using Single Instruction, Multiple Data (SIMD) hardware instructions.
Graph Partitioning
Graph partitioning is the process of dividing a large computational graph into smaller subgraphs that can be executed in parallel across multiple processors, accelerators, or devices to improve throughput and reduce latency.
Static Memory Planning
Static memory planning is a compile-time optimization that pre-allocates and reuses memory buffers for tensors within a computational graph by analyzing their lifetimes, minimizing dynamic memory allocation overhead and peak memory footprint.
In-Place Operation
An in-place operation is a computation that overwrites its input tensor's memory buffer with its output, reducing overall memory consumption by avoiding the allocation of a separate output buffer, though it destroys the input data.
Topological Sort
A topological sort is an ordering of the nodes in a directed acyclic graph (DAG) such that for every directed edge from node A to node B, node A appears before node B in the ordering, which is fundamental for scheduling operations in a computational graph.
Just-In-Time Compilation (JIT)
Just-in-time (JIT) compilation is a technique where code, such as a computational graph or kernel, is compiled during program execution rather than beforehand, allowing for optimizations based on runtime information like input shapes or hardware capabilities.
Ahead-Of-Time Compilation (AOT)
Ahead-of-time (AOT) compilation is a technique where all code, including a model's computational graph and kernels, is fully compiled to a target-specific binary before deployment, eliminating runtime compilation overhead for faster startup and predictable performance.
Peephole Optimization
Peephole optimization is a low-level compiler technique that examines a small sequence of instructions or operations (a 'peephole') and replaces them with a more efficient sequence that produces the same result, often targeting specific hardware idioms.
Canonicalization
Canonicalization is a compiler transformation that converts a computational graph or expression into a standard, simplified form to eliminate redundancy, making subsequent optimization passes and pattern matching more effective and reliable.
Shape Inference
Shape inference is the process of determining the dimensions (shape) of the output tensors for each operation in a computational graph based on the known shapes of the input tensors, which is essential for memory planning and many graph optimizations.
Kernel Auto-Tuning
Kernel auto-tuning is an automated process that searches for the optimal implementation parameters (e.g., tile sizes, number of threads) for a computational kernel on a specific hardware platform by empirically evaluating many variants against a performance model.
Polyhedral Model
The polyhedral model is a mathematical framework for the compile-time analysis and transformation of loop nests, enabling complex optimizations like loop tiling, fusion, and skewing in a unified and semantically correct manner.
Operator Dispatch
Operator dispatch is the runtime mechanism that selects the appropriate implementation (kernel) for an operation in a computational graph based on factors like the input data types, shapes, and the available hardware accelerators.
Hardware Delegate
A hardware delegate is a software component within an inference framework that offloads the execution of a subgraph or specific operations to a dedicated hardware accelerator, such as a GPU, NPU, or DSP, to improve performance and efficiency.
Profile-Guided Optimization (PGO)
Profile-guided optimization (PGO) is a compiler technique that uses data collected from representative execution runs (profiles) of a program to guide better optimization decisions, such as inlining heuristics or branch prediction, for the final compiled binary.
Data Layout Optimization
Data layout optimization involves transforming the in-memory arrangement of tensor data (e.g., from NHWC to NCHW format) to better match the access patterns of computational kernels and the underlying hardware's memory subsystem, improving cache efficiency and vectorization.
Graph Quantization Folding
Quantization folding is a graph optimization that merges or 'folds' fake quantization nodes (which simulate quantization during training) with adjacent linear operations like convolution or matrix multiplication, preparing the graph for efficient integer-only inference.
Sparse Tensor Encoding
Sparse tensor encoding is a family of data structures, such as Compressed Sparse Row (CSR) or Compressed Sparse Column (CSC), used to efficiently store and compute with tensors that contain a majority of zero values, reducing memory footprint and computation.
Graph Linting
Graph linting is the process of statically analyzing a computational graph for potential errors, inconsistencies, or suboptimal patterns, such as unsupported data types or inefficient operator sequences, before execution or compilation.
Cost Model
A cost model in a compiler is an analytical or heuristic function that estimates the computational cost (e.g., latency, memory usage) of executing a specific operation, subgraph, or schedule, guiding optimization decisions like operator fusion or graph partitioning.
ROOF-Line Model
The Roofline model is an intuitive visual performance model used to bound the maximum achievable performance of a computational kernel or loop nest based on its operational intensity and the hardware's peak compute throughput and memory bandwidth.
On-Device Model Formats
Terms related to the serialized file formats and runtime representations used to deploy compressed models to edge devices. Target: Mobile Developers, Deployment Engineers.
ONNX Runtime
ONNX Runtime is a cross-platform, high-performance inference engine for executing machine learning models in the Open Neural Network Exchange (ONNX) format.
TensorFlow Lite
TensorFlow Lite is a lightweight, open-source framework for deploying machine learning models on mobile, embedded, and edge devices, featuring a converter, interpreter, and hardware acceleration delegates.
Core ML
Core ML is Apple's machine learning framework for integrating trained models into iOS, macOS, watchOS, and tvOS applications, with optimizations for Apple Silicon and the Neural Engine.
TensorRT
TensorRT is NVIDIA's high-performance deep learning inference SDK and runtime, which optimizes models for deployment on NVIDIA GPUs through techniques like layer fusion, precision calibration, and kernel auto-tuning.
OpenVINO
OpenVINO is an open-source toolkit from Intel for optimizing and deploying deep learning inference across Intel hardware, including CPUs, GPUs, VPUs, and FPGAs.
MediaPipe
MediaPipe is a cross-platform framework from Google for building multimodal applied machine learning pipelines, offering pre-built solutions and efficient on-device inference.
PyTorch Mobile
PyTorch Mobile is an end-to-end workflow for deploying PyTorch models on iOS and Android devices, supporting TorchScript models and mobile-optimized runtime.
TFLite Micro
TFLite Micro is a port of TensorFlow Lite designed to run machine learning models on microcontrollers and other deeply embedded systems with only kilobytes of memory.
Arm NN
Arm NN is an open-source inference engine that bridges between existing neural network frameworks and Arm Cortex-A CPUs, Arm Mali GPUs, and Arm Ethos NPUs.
SNPE
The Snapdragon Neural Processing Engine (SNPE) is a Qualcomm SDK for accelerating deep neural network inference on devices powered by Snapdragon mobile platforms.
Android NNAPI
The Android Neural Networks API (NNAPI) is an Android C API designed for running computationally intensive operations for machine learning on Android devices, providing a hardware abstraction layer for accelerators.
Windows ML
Windows ML is a high-performance, on-device AI inference API built into Windows 10 and 11 that allows applications to use pre-trained machine learning models locally on a Windows device.
Qualcomm AI Engine
The Qualcomm AI Engine is a heterogeneous computing architecture within Snapdragon platforms that orchestrates AI workloads across the Hexagon processor, Adreno GPU, and Kryo CPU.
Model Serialization
Model serialization is the process of converting a trained machine learning model's architecture, weights, and configuration into a persistent file format for storage and deployment.
SavedModel
SavedModel is TensorFlow's universal serialization format for models, containing a complete TensorFlow program including trained parameters and computation graphs.
TorchScript
TorchScript is an intermediate representation of a PyTorch model that can be run independently from Python, enabling deployment in high-performance, production environments like mobile and C++.
FlatBuffers
FlatBuffers is an efficient cross-platform serialization library, used by frameworks like TensorFlow Lite, that allows direct access to serialized data without parsing/unpacking.
Model Interpreter
A model interpreter is a runtime component that loads a serialized model, maps its operations to executable kernels, manages tensor memory, and executes the inference graph.
Delegate API
A delegate API is an interface in an inference framework that allows specific operations or subgraphs to be offloaded for execution to a dedicated hardware accelerator like a GPU, DSP, or NPU.
Hardware Accelerator
A hardware accelerator is a specialized processor, such as a GPU, NPU, or DSP, designed to efficiently execute specific computational workloads, like matrix multiplications common in neural networks.
Edge TPU
The Edge TPU is Google's purpose-built application-specific integrated circuit (ASIC) designed to run high-performance, low-power machine learning inference at the edge.
Hexagon DSP
The Hexagon DSP is a digital signal processor core in Qualcomm Snapdragon SoCs, often used via the Hexagon Tensor Accelerator (HTA) for efficient, low-power execution of quantized neural networks.
Apple Neural Engine
The Apple Neural Engine (ANE) is a dedicated neural network accelerator embedded in Apple Silicon, designed to optimize on-device machine learning tasks for performance and energy efficiency.
Model Zoo
A model zoo is a repository of pre-trained machine learning models, often provided by framework vendors or the community, that are ready for inference or fine-tuning.
TFLite Converter
The TFLite Converter is a tool that transforms TensorFlow models (SavedModel, Keras, Concrete Functions) into the optimized FlatBuffer format used by the TensorFlow Lite interpreter.
ONNX Simplifier
ONNX Simplifier is a tool that applies graph optimization passes, such as constant folding and redundant node elimination, to simplify and clean up ONNX models for more efficient inference.
JIT Compilation
Just-in-Time (JIT) compilation in machine learning refers to the runtime compilation of a model's computational graph into optimized machine code for the target hardware, often trading initial latency for peak performance.
AOT Compilation
Ahead-of-Time (AOT) compilation is the process of compiling a machine learning model's computational graph into an optimized executable or library for a specific target hardware platform before runtime.
Extreme Quantization
Terms related to quantization techniques that push precision to very low bit-widths, such as binarization or ternarization. Target: ML Researchers, TinyML Engineers.
Binarization
Binarization is an extreme quantization technique that constrains neural network weights and/or activations to binary values, typically +1 and -1, enabling massive reductions in model size and enabling highly efficient bitwise operations.
Ternarization
Ternarization is a quantization method that restricts neural network weights to three possible values, commonly -1, 0, and +1, offering a compromise between the extreme efficiency of binarization and higher representational capacity.
XNOR-Net
XNOR-Net is a pioneering neural network architecture that employs binarized weights and activations, replacing costly floating-point multiplications with efficient XNOR and bit-counting operations to achieve extreme computational efficiency.
BinaryConnect
BinaryConnect is a training method for deep neural networks that uses binary weights during the forward and backward passes while maintaining high-precision gradients, enabling effective training of binarized models.
Ternary Weight Networks (TWN)
Ternary Weight Networks (TWN) are a class of neural networks where weights are quantized to ternary values {-1, 0, +1}, often with a learned layer-wise scaling factor, to balance model compression and accuracy.
DoReFa-Net
DoReFa-Net is a neural network framework that extends quantization to low-bit widths for weights, activations, and even gradients, enabling end-to-end training of models with arbitrary bit precision.
1-bit Quantization
1-bit quantization is the process of representing neural network parameters or activations using a single bit, effectively implementing binarization to achieve the highest possible compression ratio for on-device deployment.
Binary Activation
Binary activation is a technique where the output of a neural network layer's activation function is constrained to a binary state, drastically reducing memory traffic and enabling the use of bitwise logic for computation.
Straight-Through Estimator (STE)
The Straight-Through Estimator (STE) is a method used to approximate gradients through non-differentiable quantization functions during backpropagation, enabling the training of networks with discrete-valued parameters.
Scaling Factor (Alpha)
In extreme quantization, a scaling factor (often denoted alpha) is a learned or calculated multiplier applied to low-bit weights or activations to recover dynamic range and minimize the quantization error.
Quantization-Aware Training (QAT)
Quantization-Aware Training (QAT) is a process where a neural network is trained or fine-tuned with simulated quantization operations, allowing the model to adapt its parameters to the precision loss before deployment.
Post-Training Quantization (PTQ)
Post-Training Quantization (PTQ) is a compression technique that reduces the precision of a pre-trained model's weights and activations without requiring retraining, using calibration data to set quantization parameters.
Binarized Batch Normalization
Binarized batch normalization is a specialized variant of batch normalization designed to work with binary activations, typically involving the removal of scaling and bias parameters to maintain computational efficiency.
Channel-Wise Scaling
Channel-wise scaling is a quantization granularity technique where a unique scaling factor is learned or calculated for each output channel of a convolutional layer, improving accuracy for low-bit networks.
Mixed-Precision Quantization
Mixed-precision quantization is a strategy that assigns different bit-widths to different layers, channels, or weights within a neural network based on their sensitivity, optimizing the trade-off between model size and accuracy.
Uniform Quantization
Uniform quantization is a method where the quantization levels are evenly spaced across the range of values, simplifying the quantization and dequantization process at the cost of potentially higher error for non-uniform distributions.
Non-Uniform Quantization
Non-uniform quantization allocates quantization levels unevenly, often concentrating them in regions where the parameter distribution is denser, to better preserve information at very low bit-widths.
Logarithmic Quantization
Logarithmic quantization is a non-uniform method where values are quantized to powers of two, enabling the replacement of multiplications with efficient bit-shift operations during inference.
Integer-Only Inference
Integer-only inference is an execution paradigm where all computations of a quantized neural network, including activations, are performed using integer arithmetic, eliminating the need for floating-point units on target hardware.
PACT (Parameterized Clipping Activation)
PACT is a quantization-aware training method that uses a parameterized clipping activation function to learn the optimal range for quantizing activations, improving the accuracy of low-precision models.
LSQ (Learned Step Size Quantization)
Learned Step Size Quantization (LSQ) is a quantization-aware training technique that treats the quantization step size as a trainable parameter, allowing the model to jointly optimize weights and quantization scales.
AdaRound
AdaRound is a post-training quantization method that optimizes the rounding of weights to integers by learning a task-loss-aware rounding policy, significantly improving the accuracy of low-bit models without retraining.
Quantization Grid
A quantization grid defines the set of discrete, allowable values to which continuous numbers are mapped during quantization, such as the set {-1, 0, +1} for ternarization.
Bitwise XNOR
Bitwise XNOR is a fundamental logic operation used in binarized neural networks to approximate multiplication between binary values, forming the computational core of networks like XNOR-Net.
Bit-Serial Computation
Bit-serial computation is an execution strategy for low-bit models where operations are processed one bit at a time, trading latency for extreme reductions in hardware resource usage and power consumption.
Gradient Clipping
In the context of training quantized networks, gradient clipping is a technique used to limit the magnitude of gradients during backpropagation to stabilize training when using estimators like the Straight-Through Estimator.
Bit-Error Resilience
Bit-error resilience refers to the robustness of an extremely quantized model to errors in its binary or low-bit representations, which is a critical consideration for deployment on unreliable memory or in noisy environments.
Stochastic Quantization
Stochastic quantization is a method where values are rounded to quantization levels probabilistically based on their distance to the grid points, often used during training to act as a regularizer and improve model generalization.
Binary Neural Architecture Search (BNAS)
Binary Neural Architecture Search (BNAS) is the automated design of neural network architectures specifically optimized for binarized or extremely low-bit execution, seeking the best accuracy-efficiency Pareto frontier.
Compression Scheduling
Terms related to the strategies and algorithms for when and how to apply compression techniques during the training or fine-tuning lifecycle. Target: ML Engineers, Research Scientists.
Pruning Schedule
A pruning schedule is a predefined strategy that dictates the timing, rate, and criteria for removing parameters from a neural network during training or fine-tuning to achieve a target sparsity.
Iterative Magnitude Pruning
Iterative magnitude pruning is a compression technique that repeatedly removes the smallest-magnitude weights from a network, followed by fine-tuning to recover accuracy, often guided by the Lottery Ticket Hypothesis.
Lottery Ticket Hypothesis
The Lottery Ticket Hypothesis posits that within a dense, randomly-initialized neural network, there exists a sparse subnetwork (a 'winning ticket') that, when trained in isolation, can match the performance of the original network.
Structured Pruning Schedule
A structured pruning schedule is a strategy for removing entire groups of parameters, such as filters, channels, or neurons, according to a planned timeline to maintain hardware-efficient sparsity patterns.
Sparsity Distribution
Sparsity distribution refers to the planned allocation of sparsity (i.e., percentage of zeroed parameters) across different layers or components of a neural network, often based on sensitivity analysis.
Layer-Wise Sensitivity
Layer-wise sensitivity is a measure of how much a model's accuracy is affected by pruning or quantizing a specific layer, used to guide non-uniform compression schedules.
Pruning-Aware Training
Pruning-aware training is a methodology where pruning constraints or regularization are incorporated from the beginning of the training process to produce models that are inherently more amenable to sparsification.
Gradual Pruning
Gradual pruning is a scheduling strategy that incrementally increases the sparsity of a model over many training steps or epochs, allowing the network to adapt smoothly and preserve accuracy.
Knowledge Distillation Schedule
A knowledge distillation schedule defines the progression of training phases, loss weightings, and temperature settings used when transferring knowledge from a teacher model to a student model.
Quantization-Aware Training (QAT) Schedule
A quantization-aware training schedule outlines the phases of training where quantization simulation is introduced, including potential warmup, calibration, and fine-tuning stages to recover accuracy.
Mixed-Precision Training
Mixed-precision training is a scheduling technique that uses different numerical precisions (e.g., FP16, BF16, FP32) for different operations or training phases to optimize memory usage and computational speed.
Low-Rank Adaptation (LoRA) Scheduling
LoRA scheduling involves the strategic application and potential pruning of low-rank adapter matrices during fine-tuning to efficiently adapt large pre-trained models with minimal parameter overhead.
Automated Model Compression (AMC)
Automated Model Compression is a framework that uses reinforcement learning or other search algorithms to automatically determine the optimal pruning policy or quantization strategy for each layer of a network.
Neural Architecture Search (NAS) for Compression
NAS for compression is the use of neural architecture search algorithms to discover novel, efficient network architectures that are inherently small, fast, and accurate for deployment on resource-constrained devices.
Differentiable Neural Architecture Search (DNAS)
Differentiable Neural Architecture Search is a gradient-based NAS method that formulates the search for an optimal sub-network as a continuous optimization problem, often used to find efficient architectures for compression.
Compression Policy
A compression policy is a comprehensive set of rules and algorithms that govern which compression techniques (pruning, quantization) to apply, when to apply them, and with what intensity across a model's lifecycle.
Multi-Stage Compression
Multi-stage compression is a scheduling paradigm where different compression techniques (e.g., pruning, then quantization) are applied in separate, sequential phases, often with recovery fine-tuning in between.
Dynamic Network Surgery
Dynamic network surgery is an iterative compression technique that not only prunes connections but also regrows them during training based on gradient signals, allowing the network topology to evolve.
Sparse Evolutionary Training (SET)
Sparse Evolutionary Training is an algorithm that initializes a network with a sparse topology and dynamically prunes and regrows connections during training based on weight magnitudes, maintaining a fixed level of sparsity.
One-Shot Pruning
One-shot pruning is a scheduling approach where a large portion of model parameters are removed in a single step, after which the sparse model is fine-tuned, as opposed to iterative removal.
Post-Training Quantization (PTQ) Scheduling
PTQ scheduling refers to the ordered steps involved in post-training quantization, including calibration data selection, range estimation, and potential fine-tuning to minimize accuracy loss without original training.
Hardware-Aware Neural Architecture Search (HW-NAS)
Hardware-aware NAS is a search methodology that directly incorporates target hardware performance metrics, such as latency or energy consumption, into the objective function when searching for optimal compressed architectures.
Adaptive Compression
Adaptive compression is a scheduling strategy where the rate or type of compression applied is dynamically adjusted during training based on real-time feedback from performance monitors like loss or accuracy.
Pruning with Regrowth
Pruning with regrowth is a class of algorithms that periodically remove less important weights and regrow new connections in promising areas of the parameter space, exploring optimal sparse structures.
Cosine Pruning Schedule
A cosine pruning schedule is a strategy where the sparsity level is increased over time according to a cosine annealing function, providing a smooth, gradual transition to the target sparsity.
Compression-Accuracy Pareto Frontier
The compression-accuracy Pareto frontier is the set of optimal model configurations where no further compression can be achieved without sacrificing accuracy, and vice-versa, guiding scheduling decisions.
Feedback-Driven Scheduling
Feedback-driven scheduling is an adaptive approach where compression decisions (e.g., pruning rate, quantization bit-width) are continuously adjusted based on live metrics like validation loss or gradient norms.
Compression-Accuracy Tradeoff Analysis
Terms related to evaluating, profiling, and balancing the performance degradation of a model against the gains from compression. Target: ML Engineers, CTOs evaluating deployment feasibility.
Compression-Accuracy Tradeoff
The fundamental engineering compromise in model compression where reductions in model size, latency, or memory footprint are balanced against potential decreases in predictive performance.
Accuracy Drop
The measurable decrease in a model's performance on a validation or test set after a compression technique, such as quantization or pruning, has been applied.
Compression Ratio
A quantitative metric, often expressed as a ratio or percentage, that compares the size, parameter count, or FLOPs of an original model to its compressed version.
Tradeoff Curve
A graphical plot, often with accuracy on one axis and a compression metric (e.g., model size, latency) on the other, used to visualize the Pareto frontier of optimal compression configurations.
Pareto Frontier
In compression-accuracy analysis, the set of optimal points on a tradeoff curve where no other configuration can improve one metric (e.g., accuracy) without worsening another (e.g., model size).
Accuracy Recovery
The process of regaining lost model performance after compression, typically through techniques like fine-tuning or quantization-aware training.
Fine-Tuning After Compression
A post-compression training step where a compressed model is further trained on a task-specific dataset to recover or improve its accuracy.
Sensitivity Analysis
A systematic evaluation to determine which layers, channels, or parameters of a neural network are most sensitive to compression and thus most critical for preserving accuracy.
Layer-Wise Sensitivity
The measurement of how accuracy degrades when compression is applied to individual layers of a neural network, used to guide mixed-precision or heterogeneous compression strategies.
Calibration Dataset
A small, representative dataset used during post-training quantization to estimate the dynamic range (min/max values) of activations for setting quantization parameters.
Quantization Error
The numerical discrepancy introduced when converting continuous floating-point values to discrete integer representations, which is a primary source of accuracy degradation in quantization.
Model Degradation
A broad term for the negative impact of compression on model performance, encompassing accuracy drop, increased latency, or reduced robustness.
Performance Baseline
The original, uncompressed model's performance metrics (accuracy, latency, size) used as a reference point for evaluating the impact of compression techniques.
Golden Model
The reference, high-accuracy, uncompressed model against which the performance and fidelity of all compressed variants are compared.
Validation Accuracy
The primary metric used during compression-accuracy tradeoff analysis, measured on a held-out validation set to gauge generalization performance without overfitting to the test set.
Model Fidelity
The degree to which a compressed model's outputs match those of the original, uncompressed model, often measured using metrics like KL divergence or cosine similarity.
KL Divergence
Kullback–Leibler divergence, a statistical measure used in compression analysis to quantify the difference between the output probability distributions of an original model and its compressed version.
Output Divergence
The general phenomenon where the predictions or internal activations of a compressed model deviate from those of the original model.
Acceptable Loss
A predefined, application-specific threshold for maximum allowable accuracy degradation that a compressed model must meet to be considered viable for deployment.
Degradation Threshold
The maximum permissible drop in a key performance metric (e.g., top-1 accuracy) that defines the boundary of an acceptable compression-accuracy tradeoff for a given use case.
Compression Artifacts
Undesirable and often predictable errors or distortions in a model's output caused by the lossy nature of compression techniques like quantization or pruning.
Quantization Noise
The error signal introduced by the quantization process, modeled as additive noise that perturbs weights and activations, leading to potential accuracy degradation.
Bit-Width Selection
The process of choosing the optimal numerical precision (e.g., 8-bit, 4-bit) for different parts of a model to maximize compression benefits while minimizing accuracy loss.
Mixed-Precision Quantization
A quantization strategy that assigns different numerical precisions (bit-widths) to different layers, weights, or activations within a single model based on their sensitivity to compression.
Robustness Analysis
The evaluation of how model compression affects performance on challenging inputs, such as out-of-distribution data, adversarial examples, or corner cases.
Compression Benchmark
A standardized suite of models, datasets, and metrics used to objectively evaluate and compare the effectiveness of different compression techniques and tools.
Performance Profiling
The systematic measurement and analysis of a model's key metrics—such as accuracy, latency, memory usage, and power consumption—before and after compression.
On-Device Evaluation
The critical final stage of tradeoff analysis where a compressed model is benchmarked on the actual target edge hardware to measure real-world latency, power, and accuracy.
Energy-Efficient Inference
Terms related to optimizing model execution specifically to minimize power consumption on battery-constrained devices. Target: Embedded Systems Engineers, Product Architects for mobile/IoT.
Dynamic Voltage and Frequency Scaling (DVFS)
Dynamic Voltage and Frequency Scaling (DVFS) is a power management technique that dynamically adjusts a processor's operating voltage and clock frequency based on real-time computational workload to minimize energy consumption.
Power Gating
Power gating is a circuit-level technique that completely shuts off power to inactive or idle blocks of a silicon chip, eliminating both dynamic and static (leakage) power consumption in those regions.
Clock Gating
Clock gating is a power-saving technique that disables the clock signal to specific circuit blocks or registers when they are not performing useful work, thereby preventing unnecessary switching activity and reducing dynamic power consumption.
Performance-Per-Watt
Performance-per-watt is a key efficiency metric for computing systems, defined as the amount of useful computational work (e.g., inferences per second) delivered for each watt of power consumed.
Joule per Inference
Joule per inference is a direct energy efficiency metric that measures the total energy, in joules, required to perform a single forward pass (inference) of a machine learning model on a given hardware platform.
Milliwatt Budget
A milliwatt budget is a strict power consumption constraint, typically in the range of single-digit to tens of milliwatts, imposed on an edge device or subsystem to ensure operation within the energy limits of a small battery or energy harvester.
Thermal Throttling
Thermal throttling is a protective mechanism in processors that automatically reduces operating frequency and voltage when the chip's temperature exceeds a safe threshold, preventing overheating at the cost of reduced performance.
Energy-Delay Product (EDP)
The Energy-Delay Product (EDP) is a combined metric that multiplies the total energy consumed by a computation by its execution time, used to evaluate trade-offs between performance and efficiency in energy-constrained systems.
Power Profiling
Power profiling is the process of measuring and analyzing the detailed power consumption of a hardware system over time, often correlating power spikes with specific software operations or model layers during inference.
Sleep States
Sleep states are predefined low-power operating modes of a processor or system-on-chip (SoC) where non-essential components are powered down or clock-gated to minimize energy drain during periods of inactivity.
Wake-on-Inference
Wake-on-inference is an event-driven system architecture where a low-power, always-on coprocessor (e.g., a microcontroller or microNPU) monitors sensor inputs and only activates the main, higher-power AI accelerator when a specific inference trigger is detected.
Always-On Sensing
Always-on sensing refers to the continuous, low-power operation of sensors and a minimal inference pipeline (e.g., for keyword spotting) to enable devices to perceive and react to events without requiring user interaction or waking the main processor.
Event-Driven Inference
Event-driven inference is an execution paradigm where model inference is triggered only by specific, predefined external events (e.g., sensor data exceeding a threshold), rather than running continuously, to conserve energy.
Duty Cycling
Duty cycling is a power management strategy where a system periodically alternates between short active periods (for computation or sensing) and long sleep periods, reducing average power consumption by limiting active time.
Frames Per Joule (FPJ)
Frames per joule (FPJ) is an efficiency metric for vision systems that measures the number of image frames a system can process for each joule of energy consumed, combining throughput and power consumption.
Operations per Watt (OP/W)
Operations per watt (OP/W) is a hardware efficiency metric that quantifies the number of arithmetic operations (e.g., integer OPs, FLOPs) a processor can execute for each watt of power it consumes.
Energy Trace
An energy trace is a time-series log of a system's instantaneous power consumption, often captured by specialized hardware monitors, used to identify power-hungry software routines or inefficient hardware states.
Subthreshold Operation
Subthreshold operation is an ultra-low-power circuit design technique where transistors are operated at a gate voltage below their threshold voltage, drastically reducing switching energy at the cost of significantly slower computation speeds.
Near-Threshold Computing (NTC)
Near-threshold computing (NTC) is a design paradigm where digital circuits operate with a supply voltage close to the transistor threshold voltage, offering a highly favorable trade-off between energy efficiency and performance for non-latency-critical workloads.
Static Power (Leakage Power)
Static power, also known as leakage power, is the electrical power consumed by a CMOS integrated circuit when it is powered on but not actively switching, primarily due to unwanted current leakage through transistors.
Dynamic Power
Dynamic power is the electrical power consumed by a digital circuit due to the charging and discharging of capacitive loads (switching activity) during logic transitions, proportional to the square of the supply voltage, clock frequency, and capacitive load.
Battery-Aware Scheduling
Battery-aware scheduling is an algorithm that schedules computational tasks on a device by considering the current state of charge, health, and discharge characteristics of the battery to maximize operational lifetime or utility.
Intermittent Computing
Intermittent computing is a programming model and system design for devices powered by energy harvesters, where computations must be robust to frequent, unpredictable power losses, often using non-volatile memory and checkpointing.
Energy Harvesting
Energy harvesting is the process by which ambient energy from the environment (e.g., light, vibration, thermal gradients, or RF signals) is captured and converted into electrical energy to power small electronic devices, enabling battery-less or energy-autonomous operation.
Power Management Unit (PMU)
A Power Management Unit (PMU) is a dedicated hardware block or integrated circuit responsible for generating, regulating, sequencing, and controlling the supply voltages and power states of various components within an electronic system.
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us