Multi-Task Learning

The most common MTL architecture where the model shares hidden layers across all tasks while maintaining task-specific output layers. In NER, a shared Transformer encoder processes input text, while separate linear classifiers predict entity types, relation types, and coreference links. This drastically reduces the risk of overfitting on any single task by forcing the model to learn representations that generalize across multiple objectives. The shared encoder parameters are updated by the aggregated loss from all task-specific heads.

The performance of an MTL model is highly dependent on selecting complementary auxiliary tasks that share a hierarchical relationship with the main NER objective. Effective auxiliary tasks include:

• Relation Extraction: Identifying semantic links between entities, which reinforces entity boundary detection.
• Coreference Resolution: Clustering mentions that refer to the same entity, improving cross-sentence consistency.
• Part-of-Speech Tagging: Providing low-level syntactic signals that stabilize entity span prediction. Tasks that are too dissimilar can introduce negative transfer, degrading primary task performance.

Balancing the contribution of each task's loss during training is critical. Naively summing losses can cause one task to dominate gradient updates. Advanced strategies include:

• Uncertainty Weighting: Automatically learning task weights by modeling the homoscedastic uncertainty of each task's prediction as a learnable parameter.
• GradNorm: Dynamically adjusting weights to equalize the magnitude of gradients flowing from each task into the shared layers.
• Pareto Optimality: Framing MTL as a multi-objective optimization problem to find solutions where no task's performance can improve without harming another.

An alternative to hard parameter sharing that uses linear combinations of activations between task-specific subnetworks. A cross-stitch unit learns a matrix of mixing weights that determines how much each task's hidden representation is influenced by the representations of other tasks at each layer. This allows for more flexible sharing than a single shared encoder, enabling the model to learn a task-specific sharing pattern that can be more effective when tasks are only loosely related.

Incorporates a gradient reversal layer and a domain classifier as an auxiliary task to learn task-invariant features. While the shared encoder learns to perform NER, the adversarial domain classifier attempts to predict which task or domain the features originated from. The encoder is trained to maximize the domain classifier's loss, effectively stripping out task-specific signals and retaining only the underlying linguistic structure that is universally beneficial for all tasks.

A staged MTL approach where a model is first pre-trained on a broad set of related auxiliary tasks before being fine-tuned on the target NER task. For example, a model might be sequentially trained on language modeling, then syntax parsing, then entity typing, and finally fine-grained NER. This curriculum-based transfer mimics a learning progression from general to specific knowledge, often outperforming joint training when auxiliary task labels are abundant but target labels are scarce.

The foundational MTL architecture where a single shared encoder (e.g., BERT) feeds task-specific classification heads. All tasks share the same hidden layers, drastically reducing overfitting risk by forcing the model to learn a universal representation. For NER, this means the encoder simultaneously learns features useful for entity boundary detection and relation extraction, improving generalization on the primary task.

The strategic choice of secondary objectives that provide inductive bias. Effective auxiliary tasks for NER include:

• Part-of-Speech (POS) Tagging: Reinforces syntactic structure.
• Coreference Resolution: Links pronouns to entities, improving document-level consistency.
• Sentence Boundary Detection: Helps the model understand discourse units. Poorly chosen tasks introduce negative transfer, where conflicting objectives degrade primary task performance.

A dynamic loss balancing technique introduced by Kendall et al. that treats each task's loss weight as learnable homoscedastic uncertainty. Instead of manual tuning, the model learns to weigh tasks based on their intrinsic noise. For NER + Relation Extraction, this automatically down-weights noisy relation labels while prioritizing clean entity annotations, preventing one task from dominating the shared encoder.

A conflict resolution method for MTL that directly manipulates gradients before backpropagation. When the gradients of two tasks point in opposing directions, PCGrad projects each gradient onto the normal plane of the other, removing the conflicting component. This prevents the destructive interference that occurs when NER and sentiment analysis objectives pull the shared parameters in opposite directions.

A parameter-efficient alternative to full hard sharing. Instead of duplicating the entire encoder, lightweight bottleneck adapter modules are inserted between transformer layers. Each task gets its own adapter while the base model remains frozen. This enables MTL without catastrophic forgetting, allowing a single pre-trained model to serve NER, POS tagging, and dependency parsing simultaneously with minimal parameter overhead.

The primary failure mode of MTL where jointly training on multiple tasks degrades performance compared to single-task baselines. Occurs when:

• Auxiliary tasks are semantically unrelated (e.g., NER + translation).
• Dataset sizes are highly imbalanced.
• Task complexity differs significantly. Mitigation requires careful task affinity analysis—measuring pairwise transfer gains before committing to a shared architecture.