Multi-Task Learning (MTL)

The foundational MTL architecture where all tasks share a common set of hidden layers, with only task-specific output layers branching at the top.

• Mechanism: A single backbone network learns a universal representation, drastically reducing the risk of overfitting to any single task.
• Application: Effective when tasks are highly correlated, such as jointly predicting Click-Through Rate (CTR) and Conversion Rate (CVR) in advertising.
• Constraint: Performance degrades if tasks are too dissimilar due to negative transfer, where optimizing for one task hurts another.

Each task maintains its own model with its own parameters, but the distance between parameters is regularized to encourage similarity.

• Mechanism: Uses L2 distance or trace norm constraints between layers of different task-specific networks to share knowledge without forcing identical representations.
• Advantage: Provides greater flexibility than hard sharing, allowing tasks to diverge where necessary while still benefiting from cross-task information.
• Trade-off: Scales linearly with the number of tasks, increasing memory and compute requirements compared to shared-bottom architectures.

Advanced architectures like Multi-gate Mixture-of-Experts (MMoE) use learnable gating functions to dynamically route information.

• Mechanism: Multiple expert subnetworks are shared, but a task-specific gating network learns to weight the output of each expert differently for each task.
• Purpose: Explicitly mitigates negative transfer by allowing tasks to utilize different combinations of experts, even when objectives conflict.
• Example: In a retail recommender, one gate might weight an expert focused on price sensitivity for a purchase task, while another gate weights a content-affinity expert for an engagement task.

An optimization strategy that adjusts the relative importance of each task's loss function during training rather than using fixed, manually-tuned weights.

• Mechanism: Monitors the rate of change of the loss for each task and automatically increases the weight for tasks that are learning too slowly.
• Benefit: Removes the need for expensive hyperparameter tuning of loss coefficients, which is critical when balancing objectives with different scales like Log Loss for classification and Mean Squared Error for regression.
• Goal: Achieves a Pareto-optimal solution where no single task's performance can be improved without degrading another.

A training paradigm where a single-task teacher model's predictive knowledge is transferred to a multi-task student model via soft labels.

• Mechanism: A high-capacity model trained solely on a primary task (e.g., CVR) generates logits used as additional training targets for the MTL model.
• Use Case: Prevents the student model from forgetting a critical, sparse task when jointly trained with a dense, high-volume task like CTR prediction.
• Result: The student model learns richer feature representations by mimicking the teacher's confidence distribution, not just its hard predictions.

A probabilistic approach that weighs multiple loss functions by considering the homoscedastic uncertainty of each task.

• Mechanism: Derives a principled multi-task loss function by maximizing a Gaussian likelihood, where the noise parameter for each task is learned during training.
• Effect: Naturally down-weights noisy or difficult tasks and up-weights clean, high-confidence tasks without manual intervention.
• Application: Essential for combining continuous regression tasks (like Customer Lifetime Value Forecasting) with discrete classification tasks (like churn prediction) in a single model.

The foundational MTL pattern where all tasks share a common set of hidden layers before branching into task-specific towers. The shared bottom learns a universal representation of the input data, forcing the model to capture features relevant to all objectives. While parameter-efficient, this hard parameter sharing can suffer from negative transfer when tasks are loosely correlated, as conflicting gradients from different loss functions compete to update the same weights.

An advanced MTL architecture designed to mitigate negative transfer by using multiple expert subnetworks and task-specific gating networks. Each gate learns to weigh the output of different experts for its specific task, allowing the model to flexibly share knowledge where beneficial and separate where harmful. For example, a CTR task might heavily weight an expert specializing in user-item affinity, while a CVR task prioritizes an expert focused on price sensitivity.

A critical optimization challenge in MTL where the total loss is a weighted sum of individual task losses: L_total = Σ w_i * L_i. Manual tuning of weights is brittle. Advanced methods include:

• Uncertainty Weighting: Automatically adjusts weights based on each task's homoscedastic uncertainty, treating tasks as probabilistic outputs.
• Gradient Normalization (GradNorm): Dynamically balances gradients to ensure all tasks train at similar speeds, preventing one task from dominating the shared layers.

The core mechanism by which MTL improves generalization. By learning to predict conversion rate (CVR) alongside click-through rate (CTR), the model's shared representation is forced to encode deeper user intent signals beyond immediate clickbait. This inductive bias helps the CTR task avoid recommending items with high click probability but low purchase likelihood. The shared layers act as a regularizer, reducing overfitting to any single task's noise.

A soft parameter sharing approach where task-specific networks are connected through lateral connections at multiple depths. Unlike hard sharing, each task maintains its own parameters but can read from the hidden states of other tasks. This architecture, exemplified by the Progressive Layered Extraction (PLE) model, separates shared and task-specific components at each layer, allowing for more nuanced knowledge routing than single-level gating mechanisms.

Standard single-task metrics like AUC are insufficient for MTL evaluation. Practitioners must monitor:

• Per-Task AUC: Tracked independently to detect catastrophic forgetting or negative transfer.
• Gradient Cosine Similarity: Measures the alignment of gradients between tasks in the shared layers; low or negative values signal conflict.
• Task Interference Ratio: Quantifies how often a gradient step beneficial for one task degrades performance on another, guiding architecture selection.