Low-Rank Factorization

The foundational mathematical tool for low-rank factorization. SVD decomposes a weight matrix W into three matrices: W = UΣVᵀ. By retaining only the top-r singular values in Σ and truncating U and Vᵀ accordingly, the matrix is approximated as W ≈ (U√Σ)(√ΣVᵀ). This yields two factor matrices that replace the original layer, reducing parameters from mn to r(m+n). The Eckart-Young theorem guarantees this truncated SVD provides the optimal rank-r approximation under the Frobenius norm.

Low-rank factorization is applied differently depending on the layer type:

• Fully Connected Layers: Directly factorize the 2D weight matrix into two smaller matrices, inserting a linear bottleneck layer with no activation in between.
• Convolutional Layers: Exploit the four-way tensor structure. Techniques like CP decomposition or Tucker decomposition factorize the 4D kernel tensor into a sequence of smaller convolutions. For a 3×3 convolution, this often means replacing it with a 1×1 channel-reducing convolution followed by a 3×3 depthwise convolution.
• Attention Mechanisms: Factorize the large QKV projection matrices in transformer architectures to reduce the quadratic complexity of self-attention.

The choice of rank r directly controls the compression-accuracy trade-off. Key considerations:

• Energy Threshold: Retain singular values that capture 99%+ of the matrix's total energy (sum of squared singular values).
• Per-Layer Sensitivity: Not all layers tolerate equal compression. The first and last layers of a network are typically more sensitive to factorization and require higher relative ranks.
• Empirical Search: A binary search over rank values, guided by validation accuracy, is standard practice. Automated methods use Variational Bayesian Matrix Factorization to learn the optimal rank during fine-tuning.
• Spectrum Analysis: A sharp decay in singular values indicates high redundancy and suitability for aggressive rank reduction. A flat spectrum suggests the layer is already near full rank.

Direct factorization introduces an approximation error that degrades accuracy. A critical recovery step involves:

• Warm-Start Initialization: The factor matrices are initialized directly from the truncated SVD, providing a mathematically optimal starting point.
• Iterative Fine-Tuning: The factorized model is retrained for a small number of epochs on the original training data. This allows the factor matrices to adapt and compensate for the discarded singular vectors.
• End-to-End Tuning: Fine-tuning the entire factorized network jointly, rather than layer-by-layer, consistently yields higher final accuracy as downstream layers can adjust to the modified feature representations.

Low-rank factorization directly translates to hardware efficiency:

• Reduced MAC Operations: A factorized m×n matrix multiply with rank r requires r(m+n) multiply-accumulate operations instead of mn. For a 1024×1024 layer with r=64, this is an 8× reduction in MACs.
• DSP Slice Mapping: The smaller matrix dimensions map efficiently to the fixed-size DSP48 slices on Xilinx FPGAs, avoiding resource underutilization.
• Memory Bandwidth Relief: Fewer weight parameters mean fewer reads from off-chip DRAM, mitigating the memory-bound bottleneck identified by the roofline model.
• Streaming Architecture Compatibility: The sequential nature of the two smaller matrix multiplies maps naturally to deep FPGA pipelines with ping-pong buffering.

Low-rank factorization occupies a distinct niche in the model compression toolkit:

• vs. Pruning: Pruning creates sparse, irregular memory access patterns that are difficult to accelerate on hardware without specialized sparse engines. Factorization produces dense, regular matrices that achieve high utilization on standard FPGA DSP arrays and GPUs.
• vs. Quantization: Quantization reduces the bit-width of operations; factorization reduces the number of operations. The two techniques are orthogonal and composable—a factorized layer can be subsequently quantized to INT8 for multiplicative compression gains.
• vs. Knowledge Distillation: Distillation requires a pre-trained teacher model and careful hyperparameter tuning. Factorization is a direct, mathematically-grounded post-processing step with no architectural search required.

The core mathematical engine behind most low-rank factorization. SVD decomposes a weight matrix W into three matrices: U (left singular vectors), Σ (diagonal singular values), and Vᵀ (right singular vectors). By retaining only the top-k singular values and corresponding vectors, you obtain the optimal rank-k approximation that minimizes the Frobenius norm of the error.

• For a matrix of size m×n, full SVD costs O(mn²)
• Truncated SVD with k components reduces storage from mn to k(m+n+1)
• The singular values decay rate indicates how compressible a layer is

A generalization of matrix factorization to higher-order tensors, representing a large weight tensor as a chain of smaller 3-dimensional cores. Each core connects to its neighbors via a shared bond dimension or TT-rank, which controls the compression ratio.

• Particularly effective for compressing fully-connected layers with high-dimensional input spaces
• Avoids the exponential parameter growth of naive tensor representations
• Enables direct computation in the compressed format without decompression

The primary metric for evaluating the computational savings achieved by low-rank factorization. Replacing a dense matrix multiplication with two smaller factor matrices reduces the required multiply-accumulate (MAC) operations.

• Original dense layer: m×n FLOPs per input
• Factorized layer: m×k + k×n FLOPs per input
• Example: A 1024×1024 layer factored with rank 64 drops from 1,048,576 to 131,072 FLOPs — an 8× reduction
• Actual speedup depends on hardware utilization and memory bandwidth

A pre-factorization optimization that addresses the scale imbalance between consecutive layers. By absorbing scaling factors from one layer's weights into the next, it equalizes the dynamic range across channels, making the combined layers more amenable to low-rank approximation.

• Exploits the scale-equivariance property of ReLU activations
• Preserves mathematical equivalence of the original network
• Often applied before quantization or factorization to minimize accuracy loss
• Critical for models with depthwise separable convolutions

A complementary compression paradigm where a compact student model is trained to mimic the output distribution of a large teacher model. Low-rank factorization can be applied to the student's layers for additional compression.

• The teacher provides soft labels — probability distributions over classes — rather than hard labels
• The student learns to match the teacher's logits using KL-divergence loss
• Combined with factorization, this yields models that are both architecturally smaller and structurally compressed
• Temperature parameter T controls the softness of the teacher's output distribution

An alternative to post-training factorization that learns a natively sparse connectivity pattern from initialization. Unlike pruning dense networks, sparse training dynamically evolves the sparse structure during optimization.

• Algorithms like SET (Sparse Evolutionary Training) and RigL redistribute connections based on gradient magnitude
• Results in models that are both sparse and trainable from scratch
• Can be combined with low-rank structures for sparse + low-rank hybrid compression
• Eliminates the two-stage pipeline of train-then-factorize