Embedding Compression

Quantization reduces the numerical precision of embedding vectors, converting them from high-precision formats (like 32-bit floating-point) to lower-precision formats (like 8-bit integers). This directly shrinks memory footprint and can accelerate distance calculations during retrieval.

• Post-Training Quantization (PTQ): Applied after an embedding model is trained. Simple but may introduce minor accuracy loss.
• Quantization-Aware Training (QAT): The model is trained with simulated quantization, often preserving higher fidelity.
• Binary/ternary quantization: Extreme forms where values are reduced to 1-3 states, enabling highly efficient bitwise operations.

Quantization is a model compression technique that reduces the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). This directly decreases memory footprint and can accelerate inference on supported hardware.

• Post-Training Quantization (PTQ): Applied after training with minimal calibration data.
• Quantization-Aware Training (QAT): Simulates quantization during training for higher accuracy.
• Example: Converting embedding vectors from float32 to int8 can reduce their storage size by 75%.

Dimensionality reduction transforms high-dimensional data into a lower-dimensional space while preserving its essential structure. For embeddings, this reduces storage and retrieval cost.

• Principal Component Analysis (PCA): A linear technique that finds orthogonal axes of maximum variance.
• t-Distributed Stochastic Neighbor Embedding (t-SNE): A non-linear method for visualization.
• Uniform Manifold Approximation and Projection (UMAP): A modern technique for preserving both local and global structure.
• Use Case: Reducing 768-dimensional embeddings to 128 dimensions for efficient vector search.

Pruning is a neural network compression technique that removes less important weights, neurons, or entire layers to reduce model size and computational cost.

• Unstructured Pruning: Removes individual weights, creating an irregular sparse pattern.
• Structured Pruning: Removes entire channels or blocks, leading to hardware-friendly sparsity.
• Magnitude-Based Pruning: A common heuristic where weights with the smallest absolute values are removed.
• Impact: Can reduce model parameters by 90%+ with minimal accuracy loss when combined with fine-tuning.

Knowledge distillation is a model compression technique where a smaller, more efficient 'student' model is trained to mimic the behavior of a larger, more accurate 'teacher' model.

• Logits Distillation: The student learns from the teacher's softened output (logits) distribution.
• Feature Distillation: The student matches intermediate representations from the teacher's layers.
• Application: Creating compact embedding models that retain the semantic quality of large foundational models for retrieval tasks.

A sparse representation is a data format or model state where most elements are zero. This allows for significant storage and computational savings through specialized data structures and algorithms.

• Compressed Sparse Row (CSR): A standard format for storing sparse matrices.
• Application in Embeddings: Learned sparse high-dimensional vectors where only a few dimensions are active.
• Advantage: Enables efficient similarity search using inverted indices, as only non-zero dimensions need to be compared.

Product Quantization (PQ) is a high-dimensional vector compression technique that splits a vector into subvectors, quantizes each subvector separately, and represents the original vector by a short code of quantization indices.

• Mechanism: Creates a codebook for each subspace. A vector is represented by the concatenation of subvector code indices.
• Memory Savings: A 128-dimensional float32 vector (512 bytes) can be compressed to an 8-byte PQ code.
• Retrieval: Enables fast approximate nearest neighbor search at scale, as used in libraries like FAISS.