Hard Attention to the Task (HAT)

The core mechanism of HAT is the learning of task-specific, hard binary attention masks over the activations of a neural network. For each task, a separate, randomly initialized mask is trained. These masks are element-wise multiplied with the network's activations, creating a sparse, gated pathway. The hard sigmoid function and straight-through estimator are used to enable gradient-based learning of these discrete, binary values (0 or 1). A value of 1 allows information to pass, while 0 blocks it, creating a dedicated sub-network for each task.

HAT enforces strict parameter isolation by design. When a mask element is 0 for a given task, the corresponding neuron's activation is nullified. This means the gradients for that neuron's incoming weights are also zero for that task, preventing any updates. This creates sparse, non-overlapping sub-networks within the larger model. The degree of sparsity is controlled by a lifetime sparsity penalty during training, which encourages masks to use only a necessary fraction of the network's capacity, leaving resources free for future tasks.

HAT provides explicit mechanisms to balance knowledge sharing and interference:

• Forward Transfer: Enabled by allowing new tasks to attend to (use) neurons that were active in previous tasks. The new task's mask can be 1 for a neuron where an old mask was 1, allowing positive knowledge transfer.
• Backward Transfer Prevention: Crucially, the reverse is not true. If a neuron's mask is 0 for an old task, it is locked—its mask cannot be set to 1 for a new task. This prevents new tasks from overwriting or interfering with the functional pathways established for previous tasks, directly mitigating catastrophic forgetting.

To operate in a task-incremental learning setting, HAT requires a task identifier. This is typically provided via a learnable task embedding. During training and inference for a specific task, its corresponding embedding vector is fed into the network, often through a conditioning layer. This embedding selects which set of pre-learned binary masks to apply throughout the network's layers, activating only the sub-network allocated to that task. This makes the model's behavior explicitly task-dependent.

HAT differs fundamentally from popular regularization-based approaches like Elastic Weight Consolidation (EWC) or Synaptic Intelligence (SI).

• EWC/SI: Add a soft penalty to the loss function to slow down learning on important parameters. Interference is possible, and protection weakens over many tasks.
• HAT: Uses a hard, structural constraint via binary masks. It physically prevents gradient flow to protected parameters, offering stronger guarantees against forgetting. However, this comes at the cost of requiring task IDs and potentially less flexible parameter sharing.

While powerful, HAT has key constraints:

• Task Identity Required: It is designed for the task-incremental scenario, where the task ID is known at test time. It does not directly solve the more challenging class-incremental problem.
• Capacity Saturation: The network has a fixed total capacity. The lifetime sparsity hyperparameter must be set to reserve capacity for future tasks; poor planning can lead to exhaustion.
• Inference Overhead: Although the active sub-network is sparse, storing and applying all masks for all learned tasks introduces memory and slight computational overhead compared to a single, consolidated model.

Parameter isolation is a core strategy in continual learning where distinct, non-overlapping subsets of a model's parameters are allocated to different tasks. This physically prevents gradient interference and catastrophic forgetting by ensuring updates for a new task do not overwrite weights critical for old tasks. HAT implements this via learned binary masks.

• Core Principle: Dedicate parameters per task.
• Contrast with Regularization: Unlike methods that penalize changes to important weights (e.g., EWC), isolation completely protects them.
• Trade-off: Can lead to linear parameter growth with tasks if not managed sparsely.

Catastrophic forgetting is the primary problem HAT is designed to solve. It is the tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when trained on new data. This occurs due to overwriting of shared representations during gradient-based optimization.

• Mechanism: Unconstrained weight updates for Task B degrade the function learned for Task A.
• Stability-Plasticity Dilemma: Forgetting represents a failure to balance stability (retaining old knowledge) with plasticity (learning new things).
• Benchmark Impact: Measured as the drop in accuracy on earlier tasks after sequential training.

Elastic Weight Consolidation (EWC) is a foundational regularization-based continual learning algorithm. It estimates the importance of each parameter for previous tasks (using the Fisher information matrix) and adds a quadratic penalty to the loss function to slow down learning on important weights. This is a key alternative to HAT's isolation approach.

• Method: Soft constraint via loss regularization.
• Key Concept: Parameters are elastic—they can move, but important ones are anchored.
• Contrast with HAT: EWC allows shared parameters; HAT enforces strict isolation via masks.

Progressive Neural Networks are a dynamic architectural strategy for continual learning. A new, separate neural network column (a set of layers) is instantiated for each new task. Lateral connections from earlier columns to the new one allow the model to leverage previously learned features without risk of overwriting them.

• Architecture: Expands horizontally with each task.
• Parameter Growth: Parameters grow linearly, guaranteeing no forgetting.
• Relation to HAT: Both are isolation methods. HAT isolates within a fixed network via masks; Progressive Networks isolate by adding new network components.

Class-incremental learning (CIL) is one of the most challenging and practical continual learning scenarios. The model must learn new classes sequentially without access to a task identifier at test time, requiring it to distinguish between all seen classes. HAT is often evaluated in this setting.

• Key Challenge: No task ID at inference; model must auto-detect context.
• Evaluation: Average accuracy across all learned classes is the standard metric.
• HAT's Role: Task-specific masks can be selected based on input, allowing a single model to handle multiple classes.

The stability-plasticity dilemma is the fundamental theoretical challenge underlying all continual learning research. A learning system must be stable enough to retain long-term knowledge, yet plastic enough to incorporate new information. HAT addresses this by using attention masks to create stable, task-specific sub-networks within a plastic, adaptable base network.

• Stability: Resistance to forgetting.
• Plasticity: Capacity for new learning.
• Algorithmic Trade-off: Most methods, including HAT, explicitly balance these competing objectives.