Low-Rank Adaptation (LoRA)

LoRA operates by freezing the original weight matrix W of a pre-trained model and injecting a low-rank update ΔW = BA, where B and A are much smaller matrices. For a weight matrix of dimension d × k, LoRA uses a rank r that is significantly smaller than both d and k. This means that instead of updating d × k parameters, only d × r + r × k parameters are trained.

• Forward pass: h = Wx + BAx
• Initialization: A is initialized with random Gaussian values, B with zeros, so training starts from the original pre-trained model
• Inference: The learned matrices can be merged into the original weights (W + BA) with zero additional latency

In transformer architectures, LoRA is typically applied to the attention mechanism's weight matrices, specifically the query (W_q), key (W_k), value (W_v), and output (W_o) projection layers. Research has shown that adapting only the query and value matrices often yields sufficient performance while maximizing efficiency.

• Primary targets: Attention projection layers in each transformer block
• Optional targets: Feed-forward network layers for additional capacity
• Selection strategy: Applying LoRA to multiple weight types distributes the adaptation budget across the model's representational capacity
• Empirical finding: Adapting W_q and W_v with rank r=4 often matches full fine-tuning performance on language tasks

A defining characteristic of LoRA is that the learned low-rank matrices B and A can be explicitly merged into the original frozen weights after training. Because W_merged = W + BA produces a matrix of identical dimensions to the original, the deployed model has exactly the same computational footprint and inference latency as the original architecture.

• Deployment: Merge weights once, deploy a single standard model file
• Multi-tenancy: Keep a single base model and swap lightweight LoRA adapters for different tasks without duplicating the full model
• Storage efficiency: A full 175B parameter model checkpoint is ~350GB; a LoRA adapter at rank 16 is typically under 10MB

Because LoRA freezes the original pre-trained weights entirely, the model's foundational knowledge is structurally preserved. The low-rank updates learn only the residual task-specific signal required for the new domain, such as manufacturing terminology or defect taxonomies. This provides a strong implicit guard against catastrophic forgetting.

• Weight isolation: Pre-trained knowledge remains intact in frozen layers
• Task switching: Multiple LoRA adapters can be trained for different manufacturing lines or product families and swapped without interference
• Continual learning: New adapters can be trained sequentially as new defect types emerge without degrading performance on previously learned categories

The rank r of the decomposition matrices is the primary hyperparameter controlling the trade-off between adaptation capacity and parameter efficiency. Empirical studies show that extremely low ranks are surprisingly effective, with performance often saturating at r=4 to r=16 for language tasks.

• r=1: Minimal adaptation, suitable for very narrow domain shifts
• r=4-8: Sweet spot for most natural language domain adaptation tasks
• r=16-64: Higher capacity for complex multimodal or vision-language tasks
• Scaling property: Performance improvement follows a logarithmic relationship with rank, meaning doubling the rank yields diminishing returns beyond a certain threshold

LoRA introduces a scaling factor α (alpha) applied to the low-rank update: ΔW = (α/r) × BA. This factor controls the magnitude of the adaptation signal relative to the original pre-trained behavior. The α/r ratio effectively decouples the learning rate from the rank, allowing stable training across different rank configurations.

• Default practice: Set α to the initial rank value (e.g., α=16 for r=16) to maintain a scaling factor of 1.0
• Tuning strategy: Increasing α amplifies the adaptation signal, useful when the target domain differs significantly from pre-training data
• Stability: The scaling prevents the low-rank update from dominating the forward pass during early training steps

The overarching family of adaptation techniques that update only a small fraction of a model's internal weights. PEFT methods like LoRA allow massive industrial models to be customized for specific factory tasks—such as defect classification or natural language shop-floor interfaces—without the prohibitive computational cost of full fine-tuning.

• Key benefit: Reduces trainable parameters by 99%+ compared to full fine-tuning
• Storage efficiency: Each downstream task requires only megabytes of storage, not gigabytes
• Deployment agility: Multiple task-specific adapters can be hot-swapped on a single base model

An extension of LoRA that combines 4-bit quantization of the base model with low-rank adapters. QLoRA enables fine-tuning of massive 65-billion-parameter models on a single GPU with 48GB of memory, making industrial foundation model adaptation accessible without data center-scale infrastructure.

• Memory footprint: Reduces GPU memory requirements by up to 4x versus standard LoRA
• Precision strategy: Base weights stored in 4-bit NormalFloat, activations computed in 16-bit BrainFloat
• Edge enablement: Brings large model customization to factory-floor hardware profiles

A technique that combines the weights of two or more separately fine-tuned LoRA adapters into a single unified model without requiring access to the original training data. This enables the composition of separate industrial skills—such as visual defect inspection and instruction-following—into a single manufacturing copilot.

• Linear merging: Simple weighted averaging of adapter weights
• Task arithmetic: Adding and subtracting skill vectors to modulate model behavior
• Conflict resolution: Advanced methods like TIES-Merging resolve interference between competing parameter updates

A compression technique where a smaller student model is trained to replicate the behavior of a larger, more complex teacher foundation model. While LoRA adapts the original model, distillation transfers its generalization capabilities to a compact, deployable edge form factor suitable for real-time shop-floor inference.

• Teacher-student paradigm: Student learns from the teacher's output probability distribution, not just hard labels
• Complementary to LoRA: Use LoRA to create a specialized teacher, then distill into a tiny edge model
• Latency reduction: Distilled models achieve sub-10ms inference on embedded hardware

The tendency of a neural network to abruptly and completely forget previously learned information upon learning new information. LoRA mitigates this by freezing the original pre-trained weights, ensuring the base model's broad manufacturing knowledge is preserved while only the injected adapter matrices learn task-specific patterns.

• Weight isolation: Frozen base weights act as a stable knowledge anchor
• Adapter specialization: Each LoRA module captures only the delta for a specific task
• Sequential adaptation: Multiple adapters can be trained in sequence without mutual interference

A specific type of transfer learning that focuses on adapting a model trained on a source data distribution to perform well on a different but related target distribution. LoRA provides an efficient mechanism for this, such as adapting a general vision model to a specific factory's unique lighting conditions, camera angles, and product variants.

• Distribution shift: Addresses the gap between training and deployment environments
• Minimal target data: Effective with only hundreds of labeled factory-floor examples
• Continuous adaptation: Lightweight adapters can be updated as production conditions evolve