Post-Training Quantization (PTQ)

A defining feature of PTQ is its ability to quantize a model without access to the original training dataset. Instead, it uses a small, representative calibration dataset (often synthetic or unlabeled) to determine the optimal scaling factors (quantization ranges) for activations. This process, known as range estimation, is critical for minimizing accuracy loss.

• Method: Typically involves passing calibration data through the model to observe activation statistics.
• Benefit: Enables quantization where data privacy or availability is a concern, aligning with federated learning principles.

PTQ is primarily implemented in two modes based on when quantization parameters are calculated.

• Static Quantization: The most common PTQ approach. Scaling factors for both weights and activations are determined once during the calibration phase and remain fixed during inference. This allows for significant graph optimizations and is ideal for integer-only inference on microcontrollers.
• Dynamic Quantization: Scaling factors for activations are computed on-the-fly during inference based on the actual input data. This offers higher accuracy for models with highly variable activation ranges (e.g., LSTMs) but introduces runtime overhead, making it less suitable for ultra-constrained TinyML devices.

The primary engineering goal of PTQ is to enable efficient execution on target hardware. By converting 32-bit floating-point (FP32) operations to 8-bit integer (INT8) operations, PTQ unlocks several hardware-level optimizations.

• Lower Memory Bandwidth: INT8 weights and activations reduce data movement, a major power consumer.
• Faster Integer ALUs: Modern Neural Processing Units (NPUs) and microcontroller DSP extensions have dedicated, faster integer arithmetic logic units.
• Energy Reduction: Integer operations consume significantly less energy than floating-point operations, directly extending the energy budget for battery-powered devices.

PTQ can be applied at different levels of granularity, trading off simplicity for potential accuracy recovery.

• Per-Tensor Quantization: A single set of scaling factors is used for an entire tensor (e.g., all weights in a layer). This is the simplest and most hardware-friendly method.
• Per-Channel Quantization: Different scaling factors are applied to each output channel of a weight tensor. This finer granularity accounts for variation across channels and typically yields higher accuracy, especially for convolutional layers. It is supported by many modern edge accelerators.
• Layer-Specific Policies: Advanced PTQ toolchains allow different quantization policies (e.g., bit-width, per-tensor/channel) to be applied to different layers based on their sensitivity to quantization error.

PTQ is often contrasted with Quantization-Aware Training (QAT), a more involved but higher-fidelity compression technique.

• PTQ Process: Train FP32 Model → Calibrate → Quantize → Deploy. It is a post-training step.
• QAT Process: Train FP32 Model → Fine-tune with Simulated Quantization → Deploy. QAT bakes robustness to quantization into the model weights.
• Key Trade-off: PTQ is faster and requires no retraining, making it ideal for rapid deployment and federated settings where retraining is costly. QAT generally achieves higher accuracy for aggressive quantization (e.g., to 4-bits) but requires a training pipeline, labeled data, and more compute.

PTQ is rarely used in isolation. It is a foundational technique that combines synergistically with other model compression methods to achieve extreme memory footprint reduction for TinyML.

• Pruning + PTQ: A model is first pruned (via weight pruning or model sparsification) to remove redundant parameters, then quantized. The result is a small, sparse, integer model.
• Knowledge Distillation + PTQ: A large teacher model trains a smaller student model, which is then quantized via PTQ for deployment.
• Federated Learning + PTQ: In federated edge learning, a global FP32 model can be quantized via PTQ after federated averaging, creating a lightweight model for deployment back to clients, enabling efficient over-the-air updates.

Quantization-Aware Training (QAT) is a model compression technique where quantization error is simulated during the training process. Unlike PTQ, the model's forward and backward passes incorporate fake quantization nodes that mimic low-precision integer arithmetic. This allows the optimizer to adjust weights to compensate for the precision loss, typically yielding higher accuracy than PTQ but requiring a full retraining cycle.

• Key Mechanism: Uses Straight-Through Estimators (STEs) to approximate gradients through the non-differentiable quantization function.
• Trade-off: Higher accuracy potential than PTQ but requires more computational resources and access to the original training pipeline.

Low-precision arithmetic refers to performing mathematical computations using numerical formats with significantly fewer bits than standard 32-bit floating-point (FP32). This is the computational foundation for PTQ and includes formats like:

• INT8: 8-bit integers, the most common target for PTQ.
• FP16/BF16: 16-bit floating-point formats.
• INT4/INT2: Extremely low-bit formats for maximal compression.

For TinyML, moving from FP32 to INT8 can reduce memory bandwidth by 4x and enable the use of efficient integer-only units (IMAs) common in microcontrollers and Neural Processing Units (NPUs), drastically reducing power consumption.

Integer-only inference is the execution of a fully quantized model using exclusively integer arithmetic operations, eliminating floating-point calculations entirely. This is the ultimate goal of PTQ for deployment on microcontrollers (MCUs) which often lack a Floating-Point Unit (FPU).

• Requires: Quantization of both weights and activations to integers, along with the conversion of batch normalization and any residual floating-point scaling operations into fixed-point integer math.
• Benefit: Enables high-efficiency execution on ultra-low-power silicon, reducing latency and energy per inference, which is critical for battery-operated sensor nodes in federated networks.

A calibration dataset is a small, representative subset of data (typically 100-1000 unlabeled samples) used during PTQ to determine the optimal quantization parameters (scale and zero-point) for each tensor. The process involves:

• Running the pre-trained FP32 model on the calibration data to observe the dynamic range of activations.
• Applying algorithms like Min-Max, Moving Average Min-Max, or KL-Divergence to calculate parameters that minimize information loss when converting to integers.
• In federated edge learning, this data must be sourced locally on-device, posing a challenge if local data is scarce at the start of deployment (cold-start problem).

Model sparsification is a compression technique complementary to PTQ that induces zeros into the model's weight or gradient matrices. A sparse model can be stored and computed more efficiently using specialized libraries that skip zero operations.

• Combination with PTQ: Pruning weights before quantization (prune then quantize) often yields compounding benefits for model size reduction.
• Federated Relevance: Sparse updates, where clients only transmit the most significant gradient changes, are a core technique for communication-efficient federated learning, reducing the bandwidth needed per training round.

Memory footprint is the total amount of volatile (RAM) and non-volatile (Flash/ROM) memory consumed by a model. It is the primary hardware constraint for TinyML deployment. PTQ directly targets Flash (model storage) and RAM (runtime activations) reduction.

• Flash: Storing INT8 weights vs. FP32 weights yields a ~4x reduction.
• RAM: Quantizing activations to INT8 reduces the memory needed for intermediate tensors, which is often the limiting factor for model execution on MCUs with only tens of kilobytes of RAM.
• System Impact: A lower memory footprint enables more complex models to run on-device, facilitates firmware integration, and allows resources for an embedded FL runtime to manage local training.