Quantization-Aware Fine-Tuning

The core mechanism of QAFT is the insertion of fake quantization nodes into the model's computational graph during fine-tuning. These nodes simulate the effects of integer rounding and clipping in the forward pass, but maintain full-precision gradients in the backward pass. This allows the optimizer to adjust weights to compensate for the introduced quantization error.

• Forward Pass: Inputs and weights are quantized to integers (e.g., INT8) using learned scale and zero-point parameters, then dequantized back to floating-point for subsequent layers.
• Backward Pass: The straight-through estimator (STE) is typically used, allowing gradients to bypass the non-differentiable quantization function, flowing as if the operation were an identity function.

The primary objective of QAFT is to recover the accuracy degradation incurred during an initial post-training quantization (PTQ) step. While PTQ is fast and requires no gradients, it can lead to significant drops in model performance, especially for sensitive architectures. QAFT acts as a corrective phase, allowing the model to adapt its weights to the quantized numerical representation over a small number of epochs (typically 5-20).

This process is distinct from quantization-aware training (QAT), which introduces quantization from the beginning of training. QAFT starts from a fully trained, high-precision model and focuses on accuracy recovery, not learning from scratch.

QAFT employs a significantly lower learning rate compared to the original pre-training phase. Since the model's weights are already well-optimized, the goal is subtle adjustment, not major learning. A typical learning rate for QAFT is 1-10% of the final learning rate used in pre-training.

Common optimizers like AdamW or SGD are used, often with cosine annealing or a simple fixed schedule. The use of a small learning rate prevents the model from catastrophically forgetting its pre-trained knowledge while it learns to operate within the quantized domain.

QAFT is inherently hardware-aware. The fake quantization simulation must accurately reflect the integer arithmetic and saturation behavior of the target deployment hardware (e.g., NPU, GPU, CPU). This includes configuring for:

• Symmetric vs. Asymmetric Quantization: Choosing the scheme supported by the target backend (e.g., TensorRT often uses symmetric).
• Per-Tensor vs. Per-Channel Quantization: Setting the quantization granularity to match hardware kernel support for optimal performance.
• Activation Ranges: Using a calibration dataset to initialize activation quantization ranges before fine-tuning begins, which are then jointly optimized with the model weights.

QAFT is the final software-based optimization step before model export to a quantization backend. The output of a successful QAFT run is a model with embedded quantization parameters (scales, zero-points) and weights that are optimized for integer execution.

This model is then passed to a hardware-specific compiler or runtime:

• TensorRT: Uses QAT/QAFT models to generate optimized INT8 engines.
• TFLite: Converts models for integer-only inference on mobile and edge devices.
• OpenVINO: Optimizes models for Intel CPU, GPU, and VPU hardware. The process ensures a seamless transition from a fine-tuned PyTorch/TensorFlow model to a highly efficient production binary.

QAFT occupies a specific point in the model optimization workflow:

• vs. Post-Training Quantization (PTQ): PTQ is calibration-only, with no weight updates. QAFT adds a fine-tuning stage after PTQ for higher accuracy.
• vs. Quantization-Aware Training (QAT): QAT bakes quantization into the entire training cycle from scratch. QAFT is applied to a pre-existing pre-trained model, making it more efficient for adapting existing models.
• vs. Mixed-Precision Training: Mixed-precision training (e.g., using FP16/FP32) aims to accelerate training. QAFT aims to produce a model for integer-only inference (e.g., INT8).
• vs. Parameter-Efficient Fine-Tuning (PEFT): Techniques like LoRA can be combined with QAFT to efficiently adapt large models for quantization, updating only a small set of parameters during the recovery phase.

Quantization-Aware Training is the foundational process of simulating quantization effects during the initial training of a neural network. It inserts fake quantization nodes into the forward pass to mimic the rounding and clipping of lower-precision arithmetic, while maintaining high-precision weights for gradient updates. This allows the model to learn robust representations that are inherently resilient to the accuracy loss of subsequent quantization.

• Key Distinction from QAFT: QAT is performed from scratch or early in training, whereas Quantization-Aware Fine-Tuning applies this simulation to a pre-trained model for a limited number of epochs to recover post-quantization accuracy.

Post-Training Quantization is a model compression technique applied after a model is fully trained. It converts weights and activations from floating-point (e.g., FP32) to lower-bit integers (e.g., INT8) without any retraining. A small calibration dataset is used to determine the optimal quantization scale and zero-point parameters by observing activation ranges.

• Relationship to QAFT: PTQ is often the first step that introduces accuracy loss. QAFT is then employed as a corrective measure to fine-tune the quantized model and recover that lost accuracy, making it a common follow-up to PTQ.

Fake Quantization, or quantization simulation, is the core mechanism enabling both QAT and QAFT. It involves inserting special operations into the model's computational graph that:

• Round and clip floating-point values during the forward pass as if they were being quantized to integers.
• Store and use the original high-precision values for the backward pass and weight updates.
• This allows the model to adapt its weights to the expected noise and distortion of true quantization, minimizing final quantization error.

Frameworks like TensorFlow's tf.quantization and PyTorch's torch.ao.quantization provide APIs for fake quantization.

Quantization granularity defines the scope over which a single set of quantization parameters (scale and zero-point) is applied. The choice of granularity is a critical hyperparameter in QAFT, trading off between accuracy and computational overhead.

• Per-Tensor: One set of parameters for an entire tensor. Simple but less accurate.
• Per-Channel: Unique parameters for each output channel of a weight tensor (common for convolutions). Higher accuracy, better for QAFT.
• Per-Token/Per-Axis: Used in transformer models for activations.
• Group-wise: A balance between per-tensor and per-channel.

QAFT often employs per-channel quantization for weights to maximize accuracy recovery.

Integer-Only Inference is the ultimate deployment goal enabled by successful QAFT. It refers to executing a fully quantized model using only integer arithmetic, eliminating floating-point operations entirely. This requires:

• Weights and activations stored as integers (e.g., INT8).
• All operations, including batch normalization folding and activation functions like ReLU, reformulated for integer math.
• A compatible quantization backend (e.g., TensorRT, TFLite, OpenVINO) with optimized integer kernels.

QAFT ensures the model is optimized for this efficient execution mode on NPUs and edge devices.

A calibration dataset is a small, representative, and unlabeled subset of the training data used to determine quantization parameters. In the QAFT workflow:

1. For PTQ (Pre-QAFT): It observes the dynamic range of activations to calculate initial scale and zero-point values for static quantization.
2. During QAFT: While the primary process uses labeled data for fine-tuning, the calibration statistics may be refined.

The dataset must be representative of real inference data to avoid clipping error from mis-calibrated ranges. Typical size is 100-500 samples.