Multi-Task Learning

The foundational mechanism of MTL is inductive transfer, where knowledge gained from learning one task provides a bias or prior that improves learning on related tasks. This is achieved by forcing the model to develop shared representations in its hidden layers that capture common underlying factors across all tasks. For example, a vision model trained jointly on object classification, detection, and segmentation learns more robust, general-purpose visual features in its convolutional backbone than if trained on any single task alone.

MTL architectures are categorized by how parameters are shared:

• Hard Parameter Sharing: The most common approach. The model has a shared backbone (e.g., shared hidden layers) with task-specific heads (output layers). This forces a direct, rigid sharing of representations, reducing overfitting risk.
• Soft Parameter Sharing: Each task has its own model, but the models are regularized (e.g., via L2 distance or trace norm) to encourage their parameters to be similar. This offers more flexibility but is computationally heavier and less common in deep learning.

Learning multiple tasks acts as a powerful form of data augmentation and regularization. The noise and idiosyncrasies specific to any single task's dataset are effectively averaged out by the joint training signal from other tasks. This prevents the model from overfitting to spurious patterns in any one task, leading to a more general and robust solution. The shared representation must satisfy the constraints of multiple objectives, which regularizes the model.

MTL's success critically depends on task relatedness. Ideally, tasks should be related but not identical, sharing common underlying structures or statistical dependencies. Negative transfer occurs when jointly learning unrelated or conflicting tasks degrades performance compared to single-task learning. Mitigation strategies include:

• Task Grouping: Using algorithms to cluster related tasks.
• Gradient Modulation: Techniques like Gradient Surgery or PCGrad that project conflicting gradients to minimize interference.
• Adaptive Weighting: Dynamically adjusting the loss weight of each task during training.

MTL offers significant computational and memory efficiency compared to training separate models for each task. A single deployed model can perform multiple functions. Furthermore, it facilitates knowledge consolidation, where learning one task (e.g., depth estimation) can provide supervisory signals that aid another (e.g., surface normals estimation), as these are geometrically related. This makes MTL particularly valuable in multi-modal settings (e.g., vision-language) where a unified model processes different data types.

MTL is closely related to Parameter-Efficient Fine-Tuning (PEFT) and Continual Learning (CL).

• PEFT Integration: Methods like Multi-Task Adapters or Compacter allow a frozen pre-trained model to efficiently learn multiple downstream tasks by adding small, task-specific modules, making MTL scalable for large models.
• Contrast with CL: MTL assumes simultaneous access to all tasks during a single training phase. In contrast, CL addresses sequential task learning, fighting catastrophic forgetting. However, MTL principles inform CL techniques like experience replay, which aims to simulate a multi-task learning scenario over time.

Continual learning is a machine learning paradigm where a model learns a sequence of tasks over time, aiming to accumulate knowledge without catastrophically forgetting previously learned information. It directly addresses the stability-plasticity dilemma. Key approaches include:

• Regularization-based methods (e.g., Elastic Weight Consolidation) that penalize changes to important parameters.
• Replay-based methods that store or generate past data for rehearsal.
• Architectural methods that allocate new parameters for new tasks. Unlike multi-task learning which trains on all tasks simultaneously, continual learning is inherently sequential.

Parameter-Efficient Fine-Tuning is a family of techniques that adapts large pre-trained models to downstream tasks by updating only a small subset of parameters instead of the full model. This is crucial for making multi-task and continual learning feasible with massive models. Core methods include:

• Adapter modules: Inserting small, trainable bottlenecks into a frozen model.
• Low-Rank Adaptation (LoRA): Approximating weight updates with low-rank matrices.
• Prompt/Prefix Tuning: Optimizing continuous vectors prepended to the input. PEFT enables efficient multi-task learning by training separate, small parameter sets for each task, all sharing a common frozen backbone.

Catastrophic forgetting is the tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when trained on new data. It is the primary challenge in continual learning and a significant risk in naive sequential fine-tuning. The phenomenon occurs due to inter-task interference during gradient-based optimization, where parameters are overwritten. Mitigation strategies central to continual learning include experience replay, regularization, and PEFT methods that isolate task-specific knowledge.

These concepts measure knowledge flow between tasks in sequential learning.

• Forward Transfer: The positive influence that learning one task has on the performance or learning speed of a subsequent, related task. It indicates that previously learned representations or skills are beneficial for new tasks.
• Backward Transfer: The influence that learning a new task has on the performance of a previously learned task. It can be positive (improvement via refinement) or negative (interference leading to forgetting). In multi-task learning, positive transfer is a primary goal, while in continual learning, managing backward transfer (minimizing negative, maximizing positive) is critical.

These are key continual learning scenarios defined by the availability of task identifiers.

• Task-Agnostic Learning: The model is not explicitly informed about task boundaries or identities during training or inference. This is the most challenging and realistic setting, requiring the model to infer task context autonomously.
• Task-Incremental Learning: Tasks are presented sequentially, and the model is provided with an explicit task identifier during both training and inference. This simplifies the problem by allowing the use of task-specific adapters or output heads. Multi-task learning typically assumes all tasks and their identifiers are known upfront.

Elastic Weight Consolidation is a foundational regularization-based continual learning method. It estimates the importance of each model parameter to previously learned tasks (using an approximation of the Fisher information matrix) and then penalizes changes to important parameters during new task training. This applies a "spring-like" regularization that anchors crucial weights. While not a PEFT method itself, EWC's philosophy of identifying and protecting critical parameters informs advanced PEFT strategies for continual learning, where task-specific adapters can be seen as explicitly isolating important changes.