Vector-Matrix Quantization (VMQ)

VMQ applies product quantization independently to both the rows and columns of a weight matrix. This dual-axis approach creates a codebook of prototype vectors for each dimension. The final quantized weight is reconstructed by looking up and combining entries from the row and column codebooks. This method exponentially increases the number of representable centroids without a linear increase in memory, enabling a much higher compression ratio than quantizing the entire matrix as a single block.

By decomposing a matrix of size m x n, VMQ can represent weights using far fewer bits. For example, using 8-bit codebooks for rows and columns can effectively represent each weight with a 16-bit index pair, achieving compression comparable to 4-bit quantization but often with better accuracy retention. This makes VMQ particularly effective for compressing the large fully-connected and attention projection layers found in transformers and other large models, where memory footprint is a primary constraint.

Unlike uniform quantization, VMQ's codebooks are learned to minimize the reconstruction error of the original weight matrix during a calibration phase. This adaptive process allows it to preserve the statistical distribution of weights crucial for model performance. The technique is often applied as a post-training quantization method, requiring only a small calibration dataset and no full retraining, though it can be integrated into quantization-aware training for optimal results.

During inference, VMQ replaces large matrix multiplications with efficient table lookups and additions. The computation Wx is decomposed into operations using the much smaller codebooks. This transformation can lead to significant speedups on hardware that benefits from reduced memory bandwidth, even though it may increase arithmetic operations. It is well-suited for deployment on specialized accelerators and edge AI chips where memory access is a bottleneck.

• Scalar Quantization (e.g., INT8): Maps each individual weight to a nearest value in a uniform or non-uniform grid. Simple but offers lower compression for a given accuracy target.
• Vector-Matrix Quantization (VMQ): Quantizes groups of weights (vectors) together, capturing correlations within rows and columns. This allows for a more efficient representation, typically achieving 2-4x higher compression than 8-bit scalar quantization at similar accuracy levels.

VMQ is a cornerstone technique for deploying large language models on smartphones and other edge devices. By compressing a model's largest layers, it enables:

• Running multi-billion parameter models in under 16GB of RAM.
• Faster loading times due to reduced model size.
• Lower power consumption from reduced memory traffic. Frameworks like Llama.cpp and MLC LLM utilize variants of VMQ (often called grouped quantization or IQ2/3/4_XS) to achieve practical inference speeds on consumer hardware.

VMQ is essential for making Neural Radiance Fields (NeRF) run at interactive frame rates. By compressing the multi-layer perceptron (MLP) weights that store scene density and color, VMQ drastically reduces the memory bandwidth required per ray query. This enables:

• On-device NeRF inference for mobile AR/VR.
• Faster training convergence for Instant Neural Graphics Primitives (Instant NGP) by reducing the parameter search space.
• Integration with hybrid renderers where a lightweight, quantized NeRF provides global illumination effects.

VMQ directly targets the memory and power constraints of mobile processors and standalone VR headsets. It compresses the neural networks used for:

• Neural Supersampling: Running an upscaling network on the device's display pipeline.
• Real-Time Denoising: Deploying a lightweight denoiser for ray-traced or path-traced previews.
• Gaze-contingent Rendering: Efficiently executing multiple conditional neural fields for foveated rendering, where different regions of the visual field are rendered with varying levels of detail.

For large-scale digital twins of factories, cities, or buildings, VMQ enables efficient storage and streaming of complex neural scene representations. Applications include:

• Progressive Loading: Streaming quantized blocks of a multi-resolution hash grid or tri-plane features as a user navigates.
• Bandwidth Reduction: Transmitting updates to a dynamic neural radiance field over constrained networks by sending only the quantized delta of changed parameters.
• Archival Compression: Storing high-fidelity 3D scans for long-term preservation with minimal storage overhead.

VMQ's structure is designed for efficient execution on modern AI accelerators like Neural Processing Units (NPUs) and GPUs with dedicated integer (INT8) math units. This involves:

• Kernel Fusion: Designing fused GPU kernels that perform dequantization and the subsequent matrix multiplication in a single pass, minimizing memory traffic.
• Cache Optimization: The predictable, blocked structure of VMQ-compressed weights improves cache locality during the batch processing of ray samples.
• Co-design with Pruning: VMQ is often applied after model pruning, quantizing the sparse weight matrices for maximum compression and speed.

VMQ facilitates the real-time rendering of scenes with variable content, which is crucial for interactive applications. This applies to:

• Conditional Neural Fields: Swapping between different object styles or lighting conditions encoded as separate codebooks within the VMQ structure.
• Dynamic Neural Radiance Fields: Efficiently updating the model that drives a deformation field to animate non-rigid objects.
• Neural Texture Baking: Compressing the neural networks that encode view-dependent material properties (neural textures) for real-time rendering in a rasterization pipeline.

For cloud-rendered experiences like volumetric video streaming or metaverse applications, VMQ reduces the operational cost per user session by:

• Lowering GPU Memory Requirements: Allowing more model instances to be batched on a single server GPU.
• Decreasing Latency: Faster kernel execution reduces end-to-end inference time.
• Enabling Smaller Instance Types: Making it feasible to run high-quality neural rendering on cost-optimized cloud instances, directly impacting the total cost of ownership (TCO) for the service provider.