Quantization Bit-Width

This is the native, full-precision format used for training most neural networks. It provides a massive dynamic range (~1.4e-45 to ~3.4e38) and high precision, essential for stable gradient calculation and convergence.

• Baseline Reference: All accuracy and size metrics for compressed models are measured against the FP32 baseline.
• Hardware Cost: Requires significant memory bandwidth and compute, often executed on GPUs or high-power CPUs with floating-point units.

A reduced floating-point format used for mixed-precision training and high-performance inference. It halves the memory footprint and can accelerate computation on hardware with native 16-bit support (e.g., NVIDIA Tensor Cores).

• BF16 (bfloat16): Truncates mantissa bits but keeps the same exponent range as FP32, reducing conversion overflow risk during training.
• FP16: A standard IEEE half-precision format. Often used for inference on edge GPUs and as an intermediate target before integer quantization.

An aggressive quantization target for memory-constrained applications, offering an 8x reduction versus FP32. It is increasingly viable for Large Language Model (LLM) deployment.

• Memory-Bound Workloads: Primarily reduces model weight storage. Activation quantization at INT4 is less common due to higher error sensitivity.
• Techniques: Often employs group-wise or channel-wise quantization with sophisticated calibration to manage error. Requires hardware with INT4 support or emulation via bit-packing.

The extreme frontier of quantization, where weights are constrained to ±1 (binary) or -1, 0, +1 (ternary). This enables ultra-efficient bitwise XNOR and popcount operations.

• Theoretical Compression: Up to 32x memory savings versus FP32.
• Research & Specialized Hardware: Accuracy degradation is significant for general networks, making it an active research area. Requires custom hardware or algorithms designed for extreme sparsity and binary math.

A strategy that assigns different bit-widths to different layers, channels, or operators within the same model. The goal is to optimize the efficiency-accuracy Pareto frontier.

• Sensitivity-Based Allocation: Layers more sensitive to quantization (e.g., attention layers in transformers) keep higher precision (FP16/INT8), while robust layers (e.g., certain embeddings) are pushed to INT4.
• Automated Search: Techniques like HAQ (Hardware-Aware Automated Quantization) use reinforcement learning or heuristics to search for the optimal per-layer bit-width configuration for a target hardware platform.

The native integer arithmetic units of the deployment hardware dictate the most efficient bit-widths. Most modern Neural Processing Units (NPUs) and mobile CPUs have optimized pipelines for 8-bit (INT8) operations. Some specialized accelerators support 4-bit (INT4) or mixed-precision modes. Selecting a bit-width that aligns with the hardware's native support avoids costly emulation and maximizes throughput.

The sensitivity of the model's task to quantization error is paramount. Mission-critical applications like medical diagnostics may require 16-bit (FP16/BF16) precision. In contrast, tasks like image classification are often robust to INT8 quantization. The acceptable accuracy drop is determined through rigorous evaluation on a validation set, establishing a compression-accuracy tradeoff curve for different bit-widths.

Bit-width directly controls the model's memory footprint and the required bandwidth for loading weights and activations.

• Model Size: Reducing precision from 32-bit float (FP32) to INT8 cuts the weight storage by 4x.
• Activation Memory: Quantizing activations reduces the size of intermediate tensors, lowering peak RAM usage.
• Energy Cost: Memory access is a dominant energy consumer; lower bit-widths reduce data movement energy. This is critical for TinyML and battery-powered devices.

Lower bit-widths enable more operations per clock cycle and higher theoretical compute density. INT8 operations can be up to 4x faster than FP32 on supported hardware due to increased parallelism. However, this speedup depends on kernel optimization and avoiding data type conversions. The goal is to keep the entire computational graph in low precision to realize end-to-end latency gains.

The method for setting quantization parameters significantly impacts the final accuracy for a given bit-width.

• Per-Tensor vs. Per-Channel: Per-channel quantization assigns a unique scale/zero-point to each output channel, often recovering accuracy lost with coarse per-tensor schemes.
• Calibration Dataset: The quality and representativeness of the data used to determine these parameters (e.g., min/max ranges) is crucial for Post-Training Quantization (PTQ).
• Static vs. Dynamic: Static quantization fixes parameters at conversion time, while dynamic quantization calculates them at runtime, offering a flexibility-accuracy trade-off.

Not all neural network layers quantize equally well. The selection of bit-width must consider operator-level support.

• Linear/Convolutional Layers: Typically quantize robustly to low bit-widths.
• Attention Mechanisms & Activations: Layers with dynamic ranges (e.g., Softmax, GELU) may require higher precision (e.g., FP16) to maintain stability.
• Mixed-Precision Quantization: A practical solution is to assign higher bit-widths to sensitive layers and lower bit-widths to others, optimizing the overall efficiency-accuracy profile. This requires compiler and runtime support.

These are the affine transformation parameters that map floating-point values to integers. The scale is a floating-point multiplier that determines the step size between integer values. The zero-point is the integer value that corresponds to the real number zero, enabling accurate representation of asymmetric data ranges. Together, they define the mapping: float_value = scale * (int_value - zero_point).

The numerical distortion introduced when a continuous range of values is mapped to a finite set of discrete levels. It is the difference between the original floating-point value and its dequantized representation. Key types include:

• Rounding Error: From mapping to the nearest quantized level.
• Clipping Error: From values outside the representable range being saturated. Lower bit-width typically increases overall quantization error, directly impacting model accuracy.

The data-driven process of determining optimal quantization parameters (scale/zero-point) for a model. A small, representative dataset (the calibration set) is passed through the model to observe the actual dynamic ranges of activations and weights. Methods include:

• Min-Max: Uses observed minimum and maximum values.
• Entropy Minimization: Finds range that minimizes information loss.
• Mean-Squared Error Minimization. Calibration is critical for Post-Training Quantization (PTQ) accuracy.

A strategy that assigns different bit-widths to different parts of a model. Instead of a uniform 8-bit or 4-bit target, sensitive layers (e.g., attention mechanisms in transformers) may keep 16-bit precision, while less sensitive layers (e.g., certain embeddings) are pushed to 4-bit. This requires sensitivity analysis to profile layer-wise error contribution, optimizing the trade-off between compression ratio and accuracy loss.

The practical implementation of reduced bit-width where weights and activations are represented as integers (e.g., INT8, INT4). This enables models to leverage hardware with native integer arithmetic logic units (ALUs), which are more power-efficient and faster than floating-point units for these operations. The full inference graph, including activation quantization, must use integers to realize the full latency and energy benefits.

Techniques that push bit-width to its theoretical minimum, often below 4-bit. This includes:

• Binary Quantization (1-bit): Weights are +1 or -1.
• Ternary Quantization (2-bit): Weights are -1, 0, or +1.
• 2-bit Quantization: Uses four levels (e.g., INT2). These methods enable massive compression and replace multiplication with efficient bitwise operations (XNOR, popcount) but require specialized algorithms and often significant accuracy recovery techniques.