Intrinsic Task Vectors

An intrinsic task vector is a compact, learned representation (e.g., 64-512 dimensions) that serves as a latent code for a specific task. This vector is the sole, dense input to a hypernetwork. By compressing task information into this small vector, the method achieves extreme parameter efficiency, as the hypernetwork's parameters are shared across all tasks, and only this small vector needs to be stored or fine-tuned per new task.

The core mechanism is a hypernetwork—a secondary neural network that takes the intrinsic task vector as input and generates the complete set of weights for a task-specific adapter module (e.g., a feed-forward network with a bottleneck). This separates the task representation (the vector) from the parameter generation logic (the hypernetwork). The base model's frozen backbone remains unchanged, while adaptation is achieved via these dynamically generated, lightweight adapters.

Unlike standard adapters where the adapter architecture (e.g., bottleneck dimension) is fixed per layer, intrinsic task vectors are architecture-agnostic. The same task vector can be used by the hypernetwork to generate parameters for adapters of different sizes or types inserted into various layers of a transformer. This provides flexibility and allows the hypernetwork to learn a mapping from task space to a diverse set of parameter spaces.

This paradigm is highly efficient for multi-task learning and continual learning scenarios. To adapt to a new task, only the small intrinsic task vector needs to be learned (or retrieved), while the hypernetwork remains fixed. This avoids catastrophic forgetting and enables rapid task switching. A library of task vectors can be maintained, and tasks can be composed via vector arithmetic (e.g., interpolation) in the latent space.

Intrinsic task vectors are a learned, low-dimensional proxy for the broader concept of task vectors, which are often constructed from the difference in weight space between fine-tuned and base models. While traditional task arithmetic operates directly on high-dimensional weight deltas, intrinsic task vectors operate on a compressed latent representation. The hypernetwork effectively performs a learned form of task arithmetic in this latent space to generate the final parameter delta.

Training involves jointly optimizing the hypernetwork and the intrinsic task vectors for a set of tasks. The hypernetwork learns a general function for generating effective adapters. During inference, for a given task, its corresponding intrinsic vector is fed to the hypernetwork to instantiate the adapter parameters on-the-fly. This introduces a small upfront computational cost for parameter generation but retains the fast inference of standard adapters thereafter.

A task vector is a mathematical representation, often derived from the difference between a fine-tuned model's weights and the base model's weights, that encodes the direction and magnitude of change needed for a specific task.

• Weight-Space Direction: Formally, for a base model with weights (\theta_b) and a model fine-tuned on task (T) with weights (\theta_T), the task vector is (\tau_T = \theta_T - \theta_b).
• Enables Task Arithmetic: These vectors can be linearly combined (e.g., added, negated) to create models for new, composite tasks, forming the basis for model editing and multi-task adaptation.

Delta tuning is the overarching paradigm of adapting a pre-trained model by learning and applying a small, task-specific parameter change (a delta), while keeping the vast majority of the base model's parameters frozen.

• Core Principle: The adapted weights (\theta') are expressed as (\theta' = \theta_b + \Delta\theta), where (\Delta\theta) is sparse, low-rank, or otherwise parameter-efficient.
• Efficiency Foundation: This approach avoids the catastrophic forgetting and high compute cost of full fine-tuning, making it the foundational concept for LoRA, (IA)^3, and adapter-based methods.

Additive parameterization is a method of representing a model's adapted weights as the sum of the original frozen weights and a learned delta matrix. It is the direct mathematical implementation of the delta tuning principle.

• Formal Definition: For a weight matrix (W_0 \in \mathbb{R}^{d \times k}), the adapted weights are (W' = W_0 + \Delta W).
• Structural Constraints: The key to efficiency is imposing structure on (\Delta W), such as enforcing it to be low-rank (as in LoRA, where (\Delta W = BA)) or sparse (as in Diff Pruning).

Modular adaptation extends a base model with small, self-contained, and often composable neural modules that are tuned for specific tasks or skills, leaving the core model unchanged.

• Composability: Modules like adapters or intrinsic task vectors can be mixed, matched, or stacked to handle complex, multi-faceted problems.
• Examples: AdapterFusion learns to combine multiple pre-trained adapters. AdapterSoup ensembles adapter parameters at inference time. This paradigm enables a single frozen backbone to host a library of specialized skills.

Conditional computation is a neural network design principle where different parts of the model are activated dynamically based on the input, enabling efficient routing and specialization.

• Dynamic Pathways: Instead of a static computational graph, the network learns to activate a sparse subset of parameters relevant to the current input.
• Relation to ITVs: Intrinsic task vectors enable a form of conditional computation at the task level. The hypernetwork conditions the generated adapter parameters on the task vector, effectively creating a specialized computational pathway for that task within the larger frozen model.