Hard Attention to the Task (HAT)

The core mechanism of HAT is a set of task-specific, hard binary attention masks applied element-wise to the activations of a shared backbone network. For each neuron, the mask is either 0 (blocked) or 1 (active). This creates a sparse, isolated sub-network pathway for each task, preventing direct interference with parameters used by other tasks.

HAT enables efficient parameter sharing across tasks through the common backbone while enforcing strict isolation via the masks. This provides a compelling trade-off:

• Efficiency: The backbone's capacity is reused, avoiding the linear parameter growth of methods like Progressive Neural Networks.
• Interference Prevention: Gradients from a new task only flow through and update the unmasked active pathway, leaving the masked (inactive) parameters for other tasks unchanged.

The binary masks are not fixed but are learned differentiable parameters. During training for a task, a sigmoid-based gate and a temperature annealing schedule allow gradients to flow through the mask parameters. A sparsity-inducing L1 penalty is applied to the mask values, encouraging most gates to close (0), promoting sparse task pathways and preserving network capacity for future tasks.

HAT's architecture explicitly manages knowledge transfer:

• Prevents Negative Backward Transfer: By isolating parameters, learning a new task cannot degrade performance on previous tasks, as their dedicated pathways are frozen.
• Allows Positive Forward Transfer: The shared backbone can encode general features. If a new task activates a similar feature subset, it benefits from pre-existing, useful representations, potentially accelerating learning.

At test time, HAT requires task identity to select the correct pre-learned binary mask. This makes it primarily suited for Task-Incremental Learning scenarios. The model applies the selected mask to the backbone, activating only the sub-network specialized for that task, ensuring prediction integrity.

While powerful, HAT has key constraints:

• Task Identity Requirement: It is not natively suited for Class- or Domain-Incremental settings where task ID is unknown at inference.
• Memory Overhead: Storing a binary mask per task per layer consumes memory, though it's more efficient than storing entire model copies.
• Capacity Saturation: The fixed backbone has finite capacity; learning too many tasks can exhaust available neurons, requiring careful mask sparsity or backbone sizing.

Parameter Isolation is a family of architectural continual learning methods that assign distinct, non-overlapping subsets of a model's parameters to different tasks. This creates a strict physical separation, completely avoiding inter-task interference at the parameter level.

• Core Mechanism: A task-specific binary mask (like in HAT) or a dedicated sub-network is activated for each task.
• Key Benefit: Provides a strong theoretical guarantee against forgetting, as old task parameters are frozen.
• Trade-off: Can lead to linear parameter growth with the number of tasks if not managed carefully. HAT is a form of parameter isolation that allows for selective parameter sharing via soft attention, offering a more parameter-efficient solution.

Elastic Weight Consolidation is a foundational regularization-based continual learning method. It mitigates catastrophic forgetting by slowing down learning on network parameters deemed important for previous tasks.

• Core Mechanism: Calculates a Fisher information matrix to estimate each parameter's importance for a learned task. A quadratic penalty is added to the loss function, making important parameters "elastic"—they can change, but at a high cost.
• Contrast with HAT: While EWC allows all parameters to be potentially updated for a new task (with penalties), HAT uses hard binary masks to completely freeze a selected subset of parameters, offering a stricter form of interference prevention. EWC is often more parameter-efficient but can be less stable over many tasks.

Gradient Episodic Memory is a rehearsal-based algorithm that stores a subset of past data in a fixed-size episodic memory. It prevents forgetting by directly constraining the optimization process.

• Core Mechanism: When computing gradients for a new task, GEM projects them to ensure they do not increase the loss on the examples stored in memory for previous tasks. This is solved as a quadratic programming problem.
• Contrast with HAT: GEM is a data-centric method requiring storage of raw or processed examples. HAT is an architecture-centric method that requires no raw data from past tasks after training, making it more suitable for strict privacy scenarios or when data storage is prohibited. GEM provides stronger theoretical guarantees on worst-case forgetting.

Progressive Neural Networks are a pioneering architectural method that freezes the entire network column for a previous task and instantiates a new column for each new task, with lateral connections from old columns to the new one.

• Core Mechanism: This design prevents forgetting by construction, as old parameters are immutable. Lateral connections enable forward transfer, allowing the new column to leverage features learned in past columns.
• Contrast with HAT: Both are architectural isolation methods. However, Progressive Nets experience linear growth in parameters and compute per task. HAT, in contrast, maintains a fixed parameter budget; the base network is shared, and only the sparse binary masks grow, making it more scalable for edge deployment with many sequential tasks.

Synaptic Intelligence is an online, regularization-based method that estimates parameter importance incrementally during training. It protects important synapses by penalizing changes proportional to their accumulated contribution to past task loss reduction.

• Core Mechanism: Tracks a per-parameter importance weight online as the sum of the gradient-over-parameter product over the training trajectory. The loss function penalizes changes to parameters with high importance.
• Contrast with HAT: SI, like EWC, is a soft-constraint method allowing all parameters to change. HAT imposes a hard constraint via binary masks. SI is computationally efficient and online, but its importance estimates can drift. HAT's masks, once learned, provide a deterministic, static pathway for each task.

Learning without Forgetting is a knowledge distillation-based approach that requires no storage of previous task data. It uses the model's own responses to new data as a distillation target to preserve old task performance.

• Core Mechanism: Before updating the model on a new task, it records the model's output (logits) for the new input data. A distillation loss term is then used to encourage the updated model to maintain similar outputs for that data, thereby preserving the decision boundaries for old tasks.
• Contrast with HAT: LwF is a functional constraint method, preserving output behavior. HAT is a structural constraint method, preserving network activation pathways. LwF is highly parameter-efficient but can struggle with dissimilar tasks. HAT provides more explicit, neuron-level control over which knowledge is preserved.