Low-Rank Factorization

Low-rank factorization is a cornerstone of Parameter-Efficient Fine-Tuning (PEFT) methods, most notably LoRA (Low-Rank Adaptation). Instead of factorizing pre-trained weights, LoRA injects trainable low-rank matrices ΔW = A * B alongside frozen original weights. This allows efficient adaptation to new tasks with a tiny fraction of trainable parameters, avoiding catastrophic forgetting and maintaining the base model's general knowledge.

Factorization introduces an approximation error, as the product A*B cannot perfectly reconstruct the original matrix W. The key engineering trade-off is between the compression ratio (determined by rank r) and task performance. A lower rank increases compression but may degrade accuracy. Techniques to mitigate loss include:

• Training the factorized matrices from scratch or fine-tuning them.
• Selective factorization of only certain layers (e.g., attention layers in transformers).
• Rank search to find the optimal rank per layer.

Beyond storage savings, low-rank factorization can accelerate inference. The sequential multiplication A * (B * x) is often faster than the single large multiplication W * x, especially when r is small. However, this benefit depends on hardware and library optimizations for small matrix operations. It also reduces communication overhead in distributed training scenarios like federated learning, where smaller parameter updates are transmitted.

Low-rank factorization is often combined with other methods in a compression pipeline:

• Pruning: Removes unimportant weights first, potentially increasing the matrix's low-rank character.
• Quantization: The resulting smaller matrices (A, B) are then quantized to lower precision (e.g., INT8).
• Knowledge Distillation: A factorized (student) model can be distilled from a larger (teacher) model. It is distinct from sparse representations, which aim for zeros, whereas factorization aims for a compressed dense representation.

Knowledge distillation is a model compression technique where a smaller, more efficient student model is trained to mimic the behavior of a larger, more accurate teacher model. It transfers knowledge via softened output probabilities (logits) or intermediate feature representations.

• Mechanism: The student is trained not just on ground-truth labels, but also to match the teacher's output distribution, which contains dark knowledge about class relationships.
• Contrast with Factorization: Distillation creates a fundamentally different, smaller architecture, while factorization approximates the structure of existing large weight matrices.
• Use Case: Effective for creating compact models for deployment where the original large model is too costly.

Mixture of Experts (MoE) is a conditional computation architecture that enables large model capacity without a proportional increase in computation per input. A routing network dynamically selects a small subset of specialized expert sub-networks for each token.

• Efficiency: Only a fraction of the total parameters (e.g., 2 out of 16 experts) are activated per input, making it a form of sparse activation.
• Contrast with Factorization: MoE adds parameters but uses them sparsely; factorization reduces the parameter count of dense layers. They can be complementary.
• Example: Used in models like Mixtral 8x7B and GPT-4 to scale parameter count efficiently.

Embedding compression refers to techniques for reducing the storage size and dimensionality of dense vector embeddings, which are common in recommendation systems, NLP, and agentic memory backends.

• Methods: Includes quantization of embedding tables, dimensionality reduction (e.g., PCA), and hashing tricks.
• Application in Memory: In agentic systems, compressed embeddings allow more vectors to be stored in a fixed-size vector database, reducing retrieval latency and cost.
• Relation: While low-rank factorization compresses weight matrices, embedding compression targets the data representations stored in memory.

Deep compression is a holistic, three-stage pipeline for extreme neural network compression, famously outlined by Han et al. It sequentially applies pruning, quantization, and Huffman coding to achieve reductions of 35x to 49x in model size.

• Pipeline: 1) Prune unimportant connections. 2) Quantize weights to shared centroids (weight sharing). 3) Apply entropy coding (Huffman) to the quantized weights.
• Philosophy: Represents an aggressive, production-oriented approach to model compression. Low-rank factorization can be integrated as an alternative or complementary step to the pruning stage.
• Result: Enables large models to run on edge devices with limited memory.