Low-Rank Factorization

At its core, low-rank factorization approximates a large weight matrix W (of dimensions m × n) as the product of two smaller matrices: W ≈ A * B. The key parameter is the rank (r), which defines the inner dimension of the factors (A is m × r, B is r × n). The total parameter count reduces from m*n to *r(m+n)**. For a layer with 1024×1024 weights, using a rank of 256 reduces parameters from ~1.05M to ~524k, a 50% compression before considering further techniques like quantization.

The theoretical foundation for identifying a low-rank approximation is Singular Value Decomposition (SVD). SVD factorizes any matrix W into U * Σ * V^T, where Σ is a diagonal matrix of singular values. By keeping only the r largest singular values and their corresponding vectors in U and V, one obtains the optimal rank-r approximation in terms of Frobenius norm error. This is the benchmark for heuristic training-based factorization methods.

A dominant application is LoRA, which factorizes the weight update during fine-tuning rather than the pretrained weights themselves. For a frozen pretrained weight W₀, the update is constrained to a low-rank form: ΔW = B * A, where A and B are trainable. The forward pass becomes: h = W₀x + ΔW x = W₀x + BAx. This drastically reduces the number of trainable parameters (e.g., from 1B to 4M) while allowing efficient adaptation, making it a cornerstone of Parameter-Efficient Fine-Tuning (PEFT).

For convolutional layers with 4D weight tensors (output_channels × input_channels × height × width), factorization extends beyond simple matrices. Common approaches include:

• Channel-wise Factorization: Decompose the convolutional kernel into a depthwise convolution followed by a pointwise (1x1) convolution, as in MobileNet.
• Tucker Decomposition: Approximates the tensor via a core tensor and factor matrices for each mode.
• CP Decomposition: Expresses the tensor as a sum of rank-1 tensors. These methods directly target the computational cost of convolutions on MCUs.

Factorization can be applied via two main pipelines:

• Post-Hoc/Post-Training: Apply SVD to a trained, dense layer and replace it with two linear layers. This is fast but may incur significant accuracy drop without fine-tuning.
• Training-Aware: Integrate the low-rank structure directly into the model architecture during training or fine-tuning. The model learns optimal factors for the task, typically preserving higher accuracy. LoRA and compressed-aware training are examples of this more effective approach.

On microcontrollers, factorization primarily reduces model size (FLASH storage) and can alter inference latency and RAM usage.

• Pros: Smaller stored model, reduced memory bandwidth for weights.
• Cons: Increases sequential operations (two matrix multiplies instead of one), which may increase latency on devices without parallel compute. The intermediate activation of size r must be stored in RAM. Optimal rank r is found by profiling the trade-off on the target hardware (e.g., ARM Cortex-M) between memory savings and compute overhead.

This is the most direct application. The large feed-forward network (FFN) layers within transformer blocks, which can contain millions of parameters, are prime targets. By factorizing a weight matrix W of size [m x n] into two matrices A ([m x r]) and B ([r x n]), where the rank r is much smaller than m and n, the total parameters are reduced from m*n to r*(m+n). This drastically shrinks model size for deployment on microcontrollers, enabling larger model capacities within strict SRAM limits.

Low-Rank Factorization is the mathematical backbone of LoRA and related Parameter-Efficient Fine-Tuning (PEFT) methods. Instead of updating all weights in a dense layer during adaptation, LoRA injects trainable low-rank factor matrices A and B alongside the frozen pre-trained weights W. The update is expressed as W' = W + A*B. This approach:

• Reduces trainable parameters by thousands of times.
• Enables rapid, cost-effective domain adaptation of large models.
• Allows multiple lightweight adapters (A,B pairs) to be switched for different tasks on a single base model.

Beyond storage savings, factorization reduces computational complexity during inference. A dense matrix-vector multiplication y = Wx has O(m*n) operations. After factorization into y = A*(B*x), the operation count becomes O(r*(m+n)). When r is small, this represents a significant FLOP reduction. On microcontrollers with limited compute (e.g., Arm Cortex-M series), this directly translates to lower latency and reduced energy consumption per inference, critical for battery-powered IoT sensors and always-on devices.

While FFN layers are the primary target, factorization is also applied to other model components:

• Embedding Tables: Large vocabulary embedding matrices can be factorized to reduce their substantial memory footprint.
• Attention Projections: The query, key, and value projection matrices in the attention mechanism can be compressed, though this requires care to maintain the representation capacity needed for contextual understanding.
• Final Classifier Head: The output layer of a language model can be factorized, especially when the output vocabulary is large.

The small size of low-rank adapter weights makes them ideal for federated learning and on-device personalization. Instead of transmitting full model updates, devices can compute and send only the delta represented by the A and B matrices. This minimizes communication overhead—a major bottleneck in federated systems. On the device, a personalized adapter can be stored and executed efficiently, allowing a global base model to adapt to local user patterns without compromising privacy or overwhelming device memory.

Low-rank factorization is rarely used in isolation. It is most powerful when combined with other techniques in a compression pipeline:

1. Pruning First: Remove redundant weights via pruning, creating a sparse model.
2. Factorize Dense Blocks: Apply low-rank approximation to the remaining dense weight blocks.
3. Quantize Finally: Apply post-training quantization to the factor matrices, converting them to INT8 or INT4 formats. This combined approach achieves multiplicative reductions in model size and enables INT8 inference on factorized weights, pushing the boundaries of what's possible for TinyML deployment.

Quantization reduces the numerical precision of a neural network's weights and activations, converting them from high-precision floating-point formats (like 32-bit) to lower-precision integers (like 8-bit or 4-bit). This shrinks the model's memory footprint and enables faster integer arithmetic on hardware without native floating-point support.

• Post-Training Quantization (PTQ): Converts a pre-trained model using a calibration dataset.
• Quantization-Aware Training (QAT): Simulates quantization during training for higher final accuracy.
• INT8 Inference: A common target, using 8-bit integers for weights and activations.

Pruning removes redundant or less important parameters from a neural network to reduce its size and computational cost. The goal is to create a sparse model with minimal impact on accuracy.

• Unstructured Pruning: Removes individual weights, creating an irregular sparse pattern that requires specialized software/hardware.
• Structured Pruning: Removes entire structural components (neurons, channels, filters), yielding a smaller, dense network.
• Iterative Pruning: A cycle of pruning small amounts and fine-tuning to recover accuracy.

Knowledge Distillation is a compression technique where a smaller, efficient student model is trained to mimic the behavior of a larger, more accurate teacher model. The student learns not just from the training data labels, but from the teacher's softened output distributions (logits) and sometimes intermediate feature representations.

• Transfers learned knowledge to a deployable form.
• Often used to compress large language models into Tiny Language Models.
• Also known as Model Distillation.

Weight Clustering (or weight sharing) is a compression technique that groups similar weight values in a neural network into a smaller set of shared centroids via algorithms like k-means. The original weight matrix is replaced by a matrix of cluster indices and a small codebook of centroid values.

• Dramatically reduces storage requirements.
• During inference, weights are reconstructed from the codebook.
• Often combined with Huffman coding for further compression.

Neural Architecture Search is an automated process for designing optimal neural network architectures. It explores a vast search space of possible layer types, connections, and hyperparameters to find a network that balances accuracy with constraints like model size or latency.

• Hardware-Aware NAS: Incorporates target device metrics (latency, memory, power) directly into the search objective.
• Once-For-All Network: A 'supernet' trained once, from which many efficient subnetworks can be extracted for different deployment scenarios.

Model Sparsity refers to the proportion of zero-valued elements in a neural network's weight or activation tensors, a property induced by pruning. Exploiting sparsity can reduce computation and memory traffic, but efficiency gains depend heavily on hardware support.

• Structured Sparsity: A regular pattern of zeros (e.g., entire channels) that is easier for standard hardware to accelerate.
• N:M Sparsity: A fine-grained pattern where in every block of M weights, N are zero (e.g., 2:4 sparsity), supported by modern GPU tensor cores.