Post-Training Quantization

Converts FP32 (32-bit floating-point) weights and activations to INT8 (8-bit integer) representations. This reduces the memory footprint by up to 4x and enables the use of faster integer arithmetic instructions on modern CPUs and edge accelerators. The core trade-off is between computational efficiency and a small, acceptable drop in model accuracy.

A critical step that determines the optimal scale and zero-point parameters to map floating-point values to integer ranges. A small, representative calibration dataset is passed through the model to observe the dynamic range of activations at each layer. Common calibration methods include:

• MinMax: Uses the absolute min and max values.
• MovingAverageMinMax: Tracks a moving average of min/max across batches.
• Histogram: Uses KL divergence or MSE to find the best range that minimizes information loss.

Defines the scope at which quantization parameters are computed, directly impacting the accuracy-efficiency trade-off:

• Per-Tensor: A single scale and zero-point for an entire weight tensor. Fastest but least accurate.
• Per-Channel: Separate parameters for each output channel of a convolutional filter. The standard for quantizing weights in CNNs, offering high accuracy.
• Per-Group: Parameters for sub-groups within a channel, used in extreme low-bit quantization (e.g., INT4).

PTQ applies quantization after training is complete, requiring no retraining and minimal data. It is ideal for rapid deployment. QAT simulates quantization noise during training, allowing the model to learn to compensate for precision loss. QAT yields higher accuracy but requires full access to the training pipeline and dataset. PTQ is the default choice for edge deployment when training resources are unavailable.

Quantized INT8 models leverage specialized instruction sets for massive throughput gains:

• ARM Neon / x86 VNNI: Vectorized integer dot-product instructions on CPUs.
• NVIDIA TensorRT: Optimizes INT8 inference on GPUs with tensor cores.
• NPU and Edge TPU: Dedicated neural processing units often require or are heavily optimized for INT8 execution, making quantization a prerequisite for deployment on these accelerators.

Weight-only quantization compresses only the model weights to INT8, while activations remain in floating-point. This reduces memory and bandwidth but still requires floating-point compute. Full integer quantization converts both weights and activations to INT8, enabling the entire inference graph to execute with integer arithmetic. Full integer quantization is required for maximum acceleration on edge NPUs and microcontrollers.

A complementary compression method that removes redundant or near-zero connections within a neural network to create a sparse architecture. Unlike quantization, which reduces numerical precision, pruning eliminates entire weights.

• Magnitude pruning: removes weights below a threshold
• Structured pruning: removes entire channels or filters
• Combined with quantization, can yield 10x+ model size reduction
• Requires fine-tuning to recover accuracy after pruning

A training paradigm where a compact student model is trained to replicate the behavior of a larger, more accurate teacher model. The student learns from both hard labels and the teacher's soft probability outputs.

• Transfers predictive capability into a deployment-efficient architecture
• Often used before quantization for compound compression
• Student can be a different architecture entirely
• Preserves more accuracy than direct training of small models

A compiler optimization that merges multiple discrete neural network operations into a single kernel launch. This reduces memory bandwidth bottlenecks and kernel launch overhead on edge accelerators.

• Fuses common patterns like Conv+BatchNorm+ReLU
• Critical for maximizing NPU and GPU utilization
• Works in tandem with quantization for optimal throughput
• Reduces intermediate tensor memory allocations

A toolchain that translates a high-level model graph into an optimized, hardware-specific executable. Applies graph-level and kernel-level transformations to maximize inference throughput on target silicon.

• Examples: Apache TVM, TensorRT, OpenVINO
• Performs quantization-aware graph optimizations
• Generates code tuned for specific NPU instruction sets
• Essential bridge between training frameworks and edge hardware

The continuous monitoring process that statistically compares a deployed model's live predictions against its training baseline. Quantized models are not immune to drift and require the same vigilance.

• Tracks input distribution shifts on factory floor data
• Alerts when accuracy degrades due to changing conditions
• Triggers retraining or recalibration pipelines
• Uses metrics like Population Stability Index (PSI) and Kullback-Leibler divergence

An open-source, cross-platform inference accelerator that executes models in the Open Neural Network Exchange format. Provides hardware-agnostic optimizations for edge and cloud deployments.

• Supports INT8 and FP16 quantization execution
• Includes built-in graph optimizations and operator fusion
• Works across CPU, GPU, NPU, and FPGA backends
• Enables model portability between training and deployment