Scalar Quantization

Scalar quantization operates per dimension of a vector. Each component (e.g., a single 32-bit floating-point value) is mapped to a lower-bit representation (e.g., 8-bit integer) independently of the others. This differs from Product Quantization (PQ), which groups dimensions into subvectors and quantizes them jointly. The independence simplifies the quantization and dequantization process, as each dimension uses its own, often uniform, mapping function.

The most common form uses uniform scalar quantization. It defines a quantization codebook by establishing a minimum and maximum value for the data distribution in each dimension. The range between min and max is divided into 2^b equal-sized bins, where b is the target bit-width (e.g., 8 bits = 256 bins). Each original float value is mapped to the integer index of its corresponding bin. Dequantization approximates the original value, typically using the bin's centroid.

• Example: Mapping 32-bit floats in the range [-1.0, 1.0] to 8-bit integers (0-255). The float 0.3 might be quantized to integer 204.

The primary benefit is a direct reduction in memory footprint and memory bandwidth requirements. Reducing vectors from 32-bit floats to 8-bit integers achieves a 4x compression. This allows:

• Storing 4x more vectors in the same RAM.
• Reducing I/O latency when loading vectors from disk or across a network.
• Enabling larger datasets to fit into faster, but more limited, CPU caches, which dramatically accelerates distance computations.

Computing distances between lower-bit integers is inherently faster than between high-precision floats. Modern CPUs have specialized instructions (e.g., AVX2, AVX-512) for Single Instruction, Multiple Data (SIMD) operations on integers, allowing them to process more data per clock cycle. For common metrics like Euclidean distance (L2) or inner product, the computation can be performed directly on the quantized integers, or with minimal conversion overhead, leading to substantial query latency improvements.

Quantization is a lossy compression technique. The error introduced is bounded and predictable. The maximum quantization error for uniform quantization is half the bin width. The impact on search accuracy (recall) depends on the data distribution and the chosen bit-width. In practice, for many machine learning embeddings (e.g., from BERT, OpenAI models), 8-bit scalar quantization often results in a negligible drop in recall (e.g., < 1-2%) for k-NN search, making it a highly effective trade-off.

Scalar quantization is rarely used in isolation. It is a complementary technique applied to vectors before they are inserted into an Approximate Nearest Neighbor (ANN) index structure like HNSW or IVF. The compressed vectors are stored in the index, and distances are computed using the compressed representations. This combines the fast search properties of the graph or inverted index with the efficiency of low-bit arithmetic. Libraries like Facebook AI Similarity Search (Faiss) provide indexes like IndexScalarQuantizer and IndexIVFScalarQuantizer for this purpose.

Scalar quantization is a core technique in post-training quantization (PTQ) and quantization-aware training (QAT). It reduces the bit-depth of model weights and activations from 32-bit floating-point (FP32) to lower precision formats like 8-bit integers (INT8) or even 4-bit integers (INT4). This enables:

• Smaller model size for faster downloads and reduced storage costs.
• Lower memory bandwidth requirements during inference.
• Faster computation on hardware that has optimized integer arithmetic units (e.g., NVIDIA Tensor Cores, Intel DL Boost).
• Wider deployment of large models on resource-constrained devices.

Within vector databases, scalar quantization compresses the high-dimensional embeddings used for similarity search. By converting 32-bit or 16-bit float vectors into 8-bit integers, it delivers:

• Dramatically reduced memory footprint, often a 4x reduction versus FP32.
• Faster distance calculations (e.g., L2, inner product) using integer SIMD instructions.
• Increased effective capacity, allowing more vectors per node or lower infrastructure costs.
• It is often combined with other techniques like Product Quantization (PQ) for even higher compression ratios, though PQ is a more complex, subspace-based method.

Scalar quantization is critical for deploying models on edge devices like smartphones, IoT sensors, and microcontrollers. These environments have severe constraints on memory, power, and compute. Quantization to INT8 or lower enables:

• Battery-efficient inference by reducing data movement and leveraging low-power integer math.
• Execution on microcontrollers with limited SRAM (e.g., Arm Cortex-M series) via frameworks like TensorFlow Lite Micro.
• Real-time processing for computer vision and audio models on mobile phones.
• It is a key enabler for TinyML, where models must fit in just a few hundred kilobytes of memory.

Cloud AI accelerators are designed for quantized math. Using scalar quantization allows inference services to achieve:

• Higher throughput (QPS) by packing more model instances into GPU/TPU memory.
• Lower latency due to faster integer operations.
• Reduced cost-per-inference by improving hardware utilization.
• Major inference engines like TensorRT, ONNX Runtime, and OpenVINO provide automated quantization toolchains. For example, converting a model to INT8 can provide a 2-4x speedup on compatible NVIDIA GPUs compared to FP16, while maintaining acceptable accuracy.

While less common for final weights, scalar quantization is used during distributed training to reduce communication overhead. Techniques like gradient quantization compress the 32-bit gradients exchanged between workers (e.g., in data-parallel SGD) to 8-bit or 16-bit values. This:

• Reduces network bandwidth required for synchronization, which is often the training bottleneck.
• Accelerates multi-node and multi-GPU training sessions.
• Frameworks like PyTorch's FSDP can employ compression. The 1-bit Adam and 0/1 Adam optimizers are extreme examples using sign-based quantization for communication.

Beyond neural networks, scalar quantization is a decades-old staple in digital signal processing and multimedia codecs:

• Image Compression (JPEG): Converts DCT coefficients from a high-bit-depth representation into lower-bit integers using a quantization table.
• Audio Compression (MP3, AAC): Quantizes frequency domain coefficients, discarding perceptual less-important information.
• Sensor Data Logging: Reduces the storage needed for time-series data from industrial sensors or scientific instruments by mapping float readings to integers.
• In these domains, it is a lossy compression step that directly trades off bit-rate for fidelity or reconstruction error.

Product Quantization (PQ) is a lossy compression technique for high-dimensional vectors. Unlike scalar quantization, which processes each dimension independently, PQ splits a vector into multiple subvectors (e.g., 8 subvectors of 4 dimensions each). Each subvector is then quantized using a separate, learned codebook. Distances are approximated efficiently using precomputed lookup tables. This method achieves much higher compression ratios (e.g., 64-bit codes for 128-D vectors) but involves more complex distance approximation.

• Key Mechanism: Decomposes the vector space into a Cartesian product of low-dimensional subspaces.
• Primary Use: Extreme memory reduction for billion-scale datasets, often combined with a coarse quantizer like IVF.
• Trade-off: Introduces asymmetric distance computation for higher accuracy.

Binary Quantization is an extreme form of scalar quantization where each vector component is reduced to a single bit, typically by binarizing based on a threshold (e.g., mean or median). The resulting representation is a binary vector where distances can be computed using highly efficient bitwise operations like Hamming distance or popcount.

• Key Mechanism: Maps floating-point values to 0 or 1.
• Primary Use: Ultra-fast, memory-efficient search in high-throughput, recall-tolerant scenarios (e.g., first-stage retrieval).
• Trade-off: Significant loss of precision; often used in multi-stage search pipelines where results are re-ranked.

Asymmetric Distance Computation (ADC) is a distance calculation strategy used with compressed vector representations, notably in Product Quantization. In ADC, the query vector remains in its original, uncompressed form, while the database vectors are compressed. Distances are approximated by summing precomputed distances between the query's subvectors and the quantized database centroids.

• Key Mechanism: Avoids the error of symmetrically quantizing both the query and database vectors.
• Primary Use: To improve the accuracy of distance approximations when database vectors are heavily compressed.
• Contrast: Symmetric Distance Computation (SDC) quantizes both sides, leading to larger approximation errors.

Post-Training Quantization (PTQ) is a model compression technique that reduces the numerical precision of a trained neural network's weights and activations (e.g., from FP32 to INT8). While related, PTQ is applied to the model generating embeddings, whereas scalar quantization is applied to the embeddings after they are generated. PTQ reduces model size and accelerates inference, which is the upstream process before vectors are indexed.

• Key Mechanism: Calibrates quantization ranges using a small representative dataset without retraining.
• Primary Use: Deploying smaller, faster models for embedding generation.
• System Impact: Reduces compute cost and latency of the embedding pipeline feeding the vector database.

A Distance Metric is a mathematical function that defines a notion of distance between two points in a vector space. The choice of metric is fundamental to similarity search and interacts directly with quantization. Common metrics include:

• Euclidean Distance (L2): Measures straight-line distance. Scalar quantization directly reduces the cost of L2 calculations.
• Cosine Similarity: Measures the cosine of the angle between vectors. For normalized vectors, it correlates with inner product.
• Inner Product: Calculates the dot product. Quantization must preserve vector direction for this metric.

Quantization can distort distances, so the quantization range (clamping) must be chosen carefully to minimize error for the target metric.

Inverted File Index (IVF) is a coarse quantization method used for indexing. It partitions the vector dataset into clusters (Voronoi cells) using k-means. An inverted index maps each cluster centroid to a list of vectors in that cell. During search, the query is compared to centroids, and only vectors in the nearest nprobe cells are searched exhaustively.

• Key Mechanism: Uses clustering for non-exhaustive search, reducing the search space.
• Primary Use: Often combined with fine quantization (like PQ or scalar quantization) in a two-stage system (e.g., IVF + SQ8).
• Relation to SQ: Scalar quantization is frequently applied within each IVF list to further compress the vectors and speed up the final distance computations in the reduced candidate set.