Quantization

A one-shot compression technique applied after a model is fully trained. PTQ converts 32-bit floating-point weights (FP32) to 8-bit integers (INT8) without retraining. The process uses a calibration dataset—a small, representative sample of manufacturing data—to determine optimal clipping ranges and scaling factors. PTQ is the most common entry point for industrial deployment because it requires no access to the original training pipeline. However, accuracy degradation can occur with aggressive bit-width reduction, particularly for models handling rare defect types or subtle sensor anomalies.

A technique that simulates quantization noise during the training process itself. The forward pass uses fake quantization nodes that round values to lower precision, while the backward pass uses full-precision gradients to update weights. This allows the model to learn representations that are inherently robust to quantization error. QAT consistently outperforms PTQ in accuracy preservation, especially at ultra-low bit-widths like 4-bit (INT4). For manufacturing defect classification where false negatives carry high cost, QAT is the gold standard despite requiring full retraining access.

Two fundamental strategies for when scaling factors are computed:

• Dynamic Quantization: Weights are quantized ahead of time, but activations are quantized on-the-fly during inference. Each activation tensor gets its own scaling factor computed from its observed range. This adds minimal latency overhead and works well for models dominated by memory-bandwidth constraints, such as transformer-based natural language interfaces.

• Static Quantization: Both weights and activations are quantized offline using pre-computed scaling factors derived from the calibration dataset. This eliminates runtime computation of ranges, yielding the lowest possible inference latency—critical for real-time control loops in adaptive process control systems.

Quantization granularity defines the scope of the scaling factor:

• Per-Tensor Quantization: A single scale and zero-point for an entire weight tensor. Fastest to compute but can cause significant accuracy loss if weight distributions vary widely across channels.

• Per-Channel Quantization: A separate scale for each output channel in a convolutional or linear layer. This preserves fine-grained weight distributions and is essential for models deployed on Neural Processing Units (NPUs) that support channel-wise integer operations. Per-channel is the default recommendation for vision-based quality inspection models where spatial feature maps have heterogeneous value ranges.

Specialized algorithms designed for the extreme scale of Large Language Models and industrial foundation models with billions of parameters. GPTQ (GPT Post-Training Quantization) uses an approximate second-order method based on the Hessian matrix to quantize weights layer-by-layer while compensating for the quantization error in subsequent layers. This allows 4-bit quantization of 175B-parameter models on a single GPU. For factory-floor deployment of natural language shop-floor interfaces, GPTQ enables running powerful instruction-following models on edge hardware with limited VRAM.

The end-goal of quantization: executing the entire model computation using only integer arithmetic. This requires:

• Quantized weights stored as INT8 or INT4
• Quantized activations produced by each layer
• Integer-arithmetic-only kernels that perform matrix multiplication and convolution without any floating-point operations

Integer-only inference unlocks deployment on microcontrollers and embedded systems that lack floating-point units entirely. For TinyML applications on factory-floor vibration sensors, this enables always-on anomaly detection at microwatt power budgets. Frameworks like TensorFlow Lite Micro and CMSIS-NN provide the optimized kernel libraries for these extreme edge scenarios.