Quantization State

Bit-width specifies the number of bits used to represent each quantized value. It is the primary determinant of the compression ratio and potential accuracy loss.

• Common bit-widths include INT8, INT4, and FP16.
• Lower bit-widths (e.g., INT4) reduce memory footprint and increase inference speed but risk higher quantization error.
• In agent state monitoring, tracking bit-width per layer helps diagnose performance regressions and validate that deployed models match their intended precision configuration.

The scale factor (or quantization step size) is a floating-point value that maps the range of floating-point numbers to the range of integer values in the target bit-width.

• Calculated as: scale = (float_max - float_min) / (quant_max - quant_min).
• It is applied during the quantization operation: quantized_value = round(float_value / scale).
• A per-tensor scale factor applies one scale to an entire tensor, while per-channel scaling uses a unique scale for each output channel in a weight tensor, offering higher accuracy at the cost of more state metadata.

The zero point is an integer offset that aligns the value '0' in the quantized integer range with a specific floating-point value, often used to represent exact zero efficiently in asymmetric quantization schemes.

• Essential for operations where an exact zero has semantic meaning (e.g., padding in convolutions, ReLU activations).
• The dequantization formula becomes: float_value = scale * (quantized_value - zero_point).
• Monitoring the zero point ensures mathematical correctness during on-device inference, preventing subtle numerical drift in agent calculations.

Quantization granularity defines the scope over which quantization parameters (scale/zero point) are shared. It is a key trade-off between model accuracy and the overhead of the quantization state.

• Per-tensor: One set of parameters for an entire tensor. Low metadata overhead, but less accurate.
• Per-channel: Unique parameters for each channel in a weight tensor. Higher accuracy, especially for depthwise convolutions, but increases state size.
• Per-group: Parameters shared across blocks of values within a tensor. A balance between the two extremes.
• Agent telemetry must track granularity to audit the fidelity of the compressed model versus its floating-point counterpart.

The calibration method is the algorithm used to determine the optimal scale and zero-point values by analyzing the distribution of a representative dataset. The resulting statistics are a core part of the quantization state.

• Common methods include Min-Max, Moving Average Min-Max, and Entropy Calibration.
• The calibration process captures the dynamic range (min/max values) or a histogram of tensor activations.
• For agent observability, logging the calibration method and the resulting ranges for key layers provides reproducibility and aids in debugging accuracy drops when the input data distribution shifts.

The quantization scheme defines whether the mapping from float to integer is symmetric or asymmetric around zero. This high-level choice dictates how scale and zero point are used.

• Symmetric Quantization: Zero point is fixed at 0. Simpler and faster for compute, but inefficient if the tensor's value range is not symmetric (e.g., after a ReLU activation).
• Asymmetric Quantization: Zero point is determined by the data range. Better utilization of the integer range for asymmetric distributions, common for activations.
• The scheme is a fundamental property of the quantization state, impacting the mathematical kernels used during the agent's inference step.

Post-Training Quantization (PTQ) is a compression technique where a pre-trained, full-precision model (e.g., FP32) is converted to a lower-precision format (e.g., INT8) without retraining. The quantization state—scale factors and zero points—is calibrated using a small, representative dataset.

• Key Mechanism: Analyzes the statistical distribution (range, outliers) of weights and activations to determine optimal quantization parameters.
• Primary Use: Enables rapid deployment of smaller, faster models for inference, crucial for edge and mobile deployment.
• Trade-off: Simpler than QAT but may incur a higher accuracy loss, especially for models with sensitive activation distributions.

Quantization-Aware Training (QAT) is a fine-tuning process where the model is trained or fine-tuned with simulated quantization noise in the forward pass, allowing it to learn robust quantization state parameters.

• Key Mechanism: Uses fake quantization nodes during training to mimic the effects of integer arithmetic, enabling the optimizer to adjust weights accordingly.
• Primary Use: Achieves higher accuracy for low-precision models (INT8, INT4) compared to PTQ, essential for production models where accuracy is paramount.
• Outcome: Produces a model whose weights and the associated scale/zero-point parameters are co-optimized for the target bit-width.

Dynamic Quantization is a PTQ variant where the quantization parameters (scale, zero point) for activations are calculated per input at runtime, while weights are statically quantized ahead of time.

• Key Mechanism: Observes the runtime range of activation tensors for each inference batch, providing flexibility for inputs with varying distributions.
• Primary Use: Effective for models like LSTMs or transformers where activation ranges can vary significantly (e.g., with sequence length).
• Overhead: Introduces minor computational cost for calculating per-batch quantization parameters, traded for improved accuracy over static activation quantization.

Quantization Granularity defines the scope over which a single set of quantization state parameters (scale, zero point) is shared. It is a fundamental architectural choice balancing accuracy and computational overhead.

• Per-Tensor: One scale/zero-point per entire tensor. Most hardware-efficient but least accurate.
• Per-Channel: Unique scale/zero-point for each output channel of a weight tensor. Common for convolutional and linear layer weights, significantly improving accuracy.
• Per-Token/Per-Axis: For activations, parameters can be computed per token (sequence element) or per feature axis, offering finer control for dynamic inputs.
• Impact: Finer granularity preserves accuracy but increases the metadata overhead and can complicate kernel implementation on accelerators.

A Calibration Dataset is a small, representative set of unlabeled data used to calculate the optimal quantization state parameters (scale, zero point) in Post-Training Quantization.

• Purpose: To observe the statistical range (min/max) or distribution (e.g., using entropy) of model activations in response to real input data.
• Requirements: Typically 100-1000 samples. Must be representative of production data to avoid quantization drift where calibration mismatches cause severe accuracy loss.
• Process: The model performs inference on this dataset in FP32 mode, and the observed tensor ranges are analyzed by a calibration algorithm (e.g., MinMax, Entropy) to derive the final quantization parameters.

Quantization Error Monitoring is an observability practice that tracks the numerical discrepancy between the full-precision and quantized model outputs, a key signal for detecting performance degradation in production.

• Key Metric: Signal-to-Quantization-Noise Ratio (SQNR) measures the power of the desired signal versus the noise introduced by quantization.
• Implementation: Involves shadow inference, where a quantized model and a reference FP32 model run in parallel on a sample of production traffic, comparing outputs (logits, embeddings).
• Alerting: Spikes in quantization error can indicate distribution shift in input data, rendering the static quantization state suboptimal and triggering a recalibration or model retraining pipeline.