Multi-Task Learning

The most common MTL architecture. A shared hidden layer (or set of layers) learns a universal feature representation, while task-specific output heads branch off for each individual task. This drastically reduces the risk of overfitting on any single task by forcing the model to find a representation that captures the common structure of all tasks. In SRL, a shared BiLSTM or Transformer encoder might feed separate heads for dependency parsing and semantic role classification.

Instead of a single shared trunk, each task has its own model with its own parameters. A regularization mechanism (such as L2 distance or a trace norm) is applied to encourage the parameters of different models to be similar. This offers more flexibility than hard sharing, allowing tasks to diverge where necessary while still benefiting from cross-task information. It is particularly useful when tasks are loosely related or have different data scales.

The success of MTL hinges on choosing genuinely related auxiliary tasks. A poorly chosen task can introduce noise and hurt performance, a phenomenon known as negative transfer. Effective auxiliary tasks provide a useful inductive bias. For SRL, syntactic dependency parsing is a proven auxiliary task because predicate-argument structures are heavily constrained by syntax. The auxiliary task should share a hierarchical or statistical structure with the main task.

Balancing the contribution of each task's loss is a critical challenge. A naive sum of losses can allow a task with a larger gradient magnitude to dominate training. Advanced strategies include:

• Uncertainty Weighting: Learning a homoscedastic task-dependent uncertainty to automatically weight losses.
• GradNorm: Adjusting weights to ensure gradients from different tasks are at a similar scale.
• Dynamic Weight Averaging: Adjusting weights based on the rate of change of the loss for each task.

A specific soft-sharing architecture that learns a linear combination of activation maps from multiple single-task networks. Cross-stitch units are placed between parallel networks and learn a matrix of mixing weights. This allows the model to dynamically decide how much information from one task's feature space should be injected into another's. This is a more granular and learned form of feature fusion compared to simple hard sharing.

A precursor and complement to MTL where a model is first pre-trained on a data-rich auxiliary task (the source) and then fine-tuned on the target task. In modern NLP, this is exemplified by pre-training a language model (e.g., BERT) on a massive text corpus using a masked language modeling objective, then fine-tuning it on a specific SRL dataset. This can be seen as an extreme form of MTL where the auxiliary task is self-supervised.

The most common MTL architecture where a model's lower layers are shared across all tasks, while the top layers are task-specific. This drastically reduces the risk of overfitting by forcing the shared layers to learn a general representation useful for multiple objectives. In NLP, a shared Transformer encoder might feed separate linear classifiers for dependency parsing and semantic role labeling.

Instead of identical layers, each task has its own model with its own parameters, but the distance between parameters is regularized to encourage similarity. Techniques include L2 distance constraints or a trace norm. This offers more flexibility than hard sharing when tasks are loosely related, allowing the model to learn distinct but aligned features for predicate disambiguation versus argument classification.

A strategy where a secondary task is used not because its output is needed, but because it helps the main task learn better. For SRL, an auxiliary task might be part-of-speech tagging or predicting the presence of a predicate. The auxiliary loss is combined with the primary loss during training, providing an inductive bias that guides the shared representation toward useful linguistic features.

A technique to resolve negative transfer, where learning one task hurts performance on another. Projecting Conflicting Gradients (PCGrad) alters gradients that point in opposing directions. If the gradient for syntactic parsing conflicts with the gradient for semantic role labeling, PCGrad projects each onto the normal plane of the other, preventing destructive interference and stabilizing joint training.

A method to condition a shared model on the specific task being performed. A low-dimensional task vector is learned for each task and fed into the network, often via FiLM layers (feature-wise linear modulation) or simple concatenation. This allows a single massive model to switch behavior dynamically, enabling zero-shot generalization to new, unseen tasks by interpolating between known task embeddings.

A dynamic loss balancing method that treats each task's loss as a homoscedastic uncertainty. Instead of manually tuning fixed loss weights, the model learns a noise parameter for each task. Tasks with higher inherent noise automatically get a lower weight. This is crucial for MTL in NLP, where the optimal balance between a complex task like AMR parsing and a simpler one like token classification is non-trivial.