VeRA (Vector-based Random Matrix Adaptation)

The core innovation of VeRA is the use of shared, frozen random matrices for both the down-projection (A) and up-projection (B) steps. Unlike LoRA, where matrices A and B are trained from scratch, VeRA initializes them with random Gaussian values and keeps them frozen across all adapted layers. This drastically reduces the number of parameters that need to be stored and optimized, as these large matrices are now static, pre-defined projections.

To enable task-specific adaptation despite frozen projections, VeRA introduces two small, trainable scaling vectors per layer:

• A down-scaling vector (d) applied column-wise to the frozen matrix A.
• An up-scaling vector (b) applied row-wise to the frozen matrix B. These vectors learn to modulate the influence of each dimension in the random projection, effectively "steering" the frozen transformations. The combined update for a weight matrix W is computed as: ΔW = (A ⊙ d) (B ⊙ b), where ⊙ denotes element-wise scaling.

VeRA achieves a radical reduction in trainable parameters. For a layer with input dimension d_in and output dimension d_out, standard LoRA introduces r * (d_in + d_out) trainable parameters (for matrices A and B). VeRA, with a rank r, introduces only r + d_out parameters (for vectors d and b). For large d_in, this represents a reduction of roughly d_in / (d_in + d_out). In practice, VeRA can fine-tune a 7B parameter model with less than 200k total trainable parameters, compared to several million for LoRA.

VeRA is grounded in the theory of random projections and the Johnson-Lindenstrauss lemma. This principle states that high-dimensional vectors can be embedded into a much lower-dimensional space via a random linear map while approximately preserving pairwise distances. VeRA leverages this by using frozen random matrices A and B as a fixed, low-dimensional basis. The trainable vectors d and b then learn to perform the specific linear combination within this basis that is required for the task, proving that a high degree of adaptation is possible with an extremely sparse parameterization.

Empirical results show VeRA can match or approach the performance of LoRA on many NLP benchmarks while using far fewer parameters. However, this comes with trade-offs:

• Slower Convergence: VeRA often requires more training steps or epochs to reach performance parity with LoRA, as the adaptation is more constrained.
• Rank Sensitivity: The choice of rank r remains important, but the impact of the frozen random initialization can introduce some variance in final performance across runs.
• Use Case: VeRA is particularly compelling in highly resource-constrained scenarios (e.g., edge devices, multi-task serving) where storing or swapping many adapter weights is prohibitive.

VeRA sits within the broader landscape of PEFT techniques:

• Vs. LoRA: A direct descendant, replacing learned low-rank matrices with scaled random ones.
• Vs. (IA)³: Similar in spirit to methods that learn scaling vectors for existing weights, but VeRA applies scaling to injected random projections.
• Vs. LoRA-FA: LoRA-FA freezes the down-projection matrix A but learns B. VeRA freezes both A and B, pushing the concept of frozen structure further.
• Vs. BitFit: Both are extremely parameter-efficient, but BitFit trains only bias terms, while VeRA trains scaling vectors for structured random transformations, often yielding stronger performance.

VeRA is engineered for scenarios where GPU VRAM is the primary bottleneck. By sharing frozen random projection matrices (A and B) across all layers and learning only tiny scaling vectors (b and d), it reduces trainable parameters to ~0.1% of the base model, compared to ~0.5-1% for standard LoRA. This enables fine-tuning of 70B+ parameter models on consumer-grade hardware (e.g., a single 24GB GPU) where even LoRA would be infeasible, effectively democratizing access to large-scale model customization.

The architecture of VeRA is ideal for efficiently managing many specialized tasks from one foundation model. Because the large random matrices are shared and frozen, organizations can store a single copy of these matrices alongside the base model. For each new task, only the unique, minuscule scaling vectors need to be saved and swapped. This creates a highly efficient modular adaptation system where deploying a new specialized model involves loading a set of vectors a few hundred kilobytes in size, rather than gigabytes of separate adapter weights.

VeRA's design aligns perfectly with decentralized and edge computing paradigms. In federated learning, the small size of the scaling vectors minimizes the bandwidth required to transmit client updates to a central server. For on-device fine-tuning on smartphones or IoT devices, the memory footprint of the active training process is drastically reduced. The frozen random projections can be pre-loaded, and the device only needs to compute gradients for and update the lightweight vectors, making continuous, privacy-preserving personalization on edge devices a practical reality.

Researchers and developers can use VeRA to accelerate the experimental phase of model adaptation. The significantly reduced parameter count leads to faster training cycles and lower compute costs per experiment. This allows for rapid iteration on:

• Task formulation and prompt engineering
• Dataset curation and validation strategies
• Hyperparameter sweeps for the scaling vectors VeRA acts as a low-cost probe to validate the viability of a fine-tuning task before potentially committing to a more expressive (but costly) method like full fine-tuning or standard LoRA.

The low-rank bottleneck inherent to LoRA-based methods provides a form of regularization. VeRA extends this by further constraining the adaptation space through fixed random projections. This strong structural prior helps prevent the model from memorizing noise in small, domain-specific datasets. By learning only how to scale fixed random features, VeRA is less prone to catastrophic overfitting than methods with more trainable parameters, making it a robust choice for fine-tuning with only hundreds or a few thousand examples.

VeRA serves as a compelling testbed for research into the fundamental mechanisms of adaptation in large models. Its use of fixed random bases raises questions about the intrinsic dimensionality of task-specific knowledge. Researchers utilize VeRA to study:

• The minimum viable parameterization for effective transfer learning.
• The role of random projections as a universal feature basis for adaptation.
• Task arithmetic and model editing with ultra-lightweight vectors. It provides a clean, minimalist framework for probing how and where task-specific information is encoded within neural networks.

Low-Rank Adaptation (LoRA) is the foundational PEFT method that VeRA builds upon. It approximates a model's weight update ΔW as the product of two trainable low-rank matrices, A and B, where ΔW = BA. This factorization drastically reduces the number of trainable parameters compared to full fine-tuning. LoRA's core innovation is the low-rank hypothesis, which posits that weight updates during adaptation have a low "intrinsic rank," allowing for effective learning with minimal parameters. It is widely applied to transformer attention blocks.

In LoRA and related methods, the rank (r) is the critical hyperparameter defining the inner dimension of the low-rank matrices. It controls:

• Expressivity: A higher rank allows the adapter to represent more complex updates but increases parameters.
• Efficiency: A lower rank maximizes parameter reduction but may limit adaptation capacity.
• Bottleneck: This dimension creates the low-rank bottleneck that is central to LoRA's efficiency. VeRA takes this further by fixing the projection matrices and only learning scaling vectors, decoupling the rank from the number of trainable parameters.

DoRA is an advanced PEFT method that refines LoRA by decomposing a pre-trained weight matrix W into magnitude (m) and direction (V) components, where W = m V/||V||_c. It then applies LoRA specifically to the direction component V, while learning a separate scaling vector for magnitude. This separation allows for more precise and stable fine-tuning. VeRA shares a conceptual link with DoRA through its focus on learning scaling factors, but VeRA applies this to shared, frozen random matrices rather than a decomposed direction matrix.

LoRA-FA is a LoRA variant where the down-projection matrix A is initialized randomly and frozen, and only the up-projection matrix B is trained. This halves the number of trainable parameters compared to standard LoRA and reduces the computational graph. VeRA extends this concept of fixed random projections further. While LoRA-FA freezes only A, VeRA freezes both the A and B matrices, using them as fixed, shared random bases. Learning then occurs solely in the scaling vectors applied to these bases.

This is the broad category of PEFT methods that insert small, trainable modules (adapters) into a frozen pre-trained model. Key characteristics include:

• Architectural Injection: Adapters are typically inserted between layers (e.g., after the feed-forward network in a transformer).
• Bottleneck Design: They often use a down-project-up-project structure to create a parameter bottleneck.
• Modularity: Adapters can be easily swapped or combined. VeRA is an adapter method, but its adapters are composed of fixed random matrices and learnable vectors, representing an extreme in parameter reduction within this family.

Parameter efficiency is the core design goal measured by the fraction of a model's total parameters that are updated or added during adaptation. It is a key metric for comparing PEFT methods.

• Standard LoRA: Typically trains 0.1% to 1% of total parameters.
• VeRA: Aims for an order-of-magnitude further reduction, often training < 0.01% of parameters by leveraging shared random bases.
• Trade-off: The pursuit of extreme parameter efficiency, as in VeRA, involves a fundamental trade-off between reduction in trainable parameters and the representational capacity of the adapter, often requiring careful initialization and scaling.