Architectural Methods

This foundational strategy ensures zero parameter overlap between tasks by design. It allocates distinct, non-overlapping subsets of the model's weights to each task, creating a strict functional separation. This is often implemented via task-specific subnetworks, masks, or entirely separate model columns. The primary advantage is the theoretical guarantee against forgetting, as gradients from a new task cannot modify parameters assigned to old tasks. The trade-off is linear parameter growth with the number of tasks, which can become computationally inefficient for long task sequences.

A seminal architectural method that instantiates a new, task-specific neural network column for each new task. Previous columns are frozen to preserve their knowledge. To enable feature reuse and transfer learning, the new column receives lateral connections from all preceding columns, allowing it to build upon previously learned representations.

• Key Mechanism: Column freezing + lateral connections.
• Advantage: Perfect retention of old task performance.
• Limitation: Parameter count grows quadratically due to lateral connections, making scaling to many tasks resource-intensive.

HAT is a sophisticated parameter isolation method that learns binary attention masks over network units (e.g., neurons, attention heads) for each task. During training and inference for a specific task, only units whose mask is 'on' are active. The method ensures zero gradient flow to parameters assigned to other tasks by using a hard, non-differentiable gating function during the forward pass and a soft, differentiable approximation during the backward pass for learning the masks. This allows for highly efficient, sparse activation patterns tailored to each task.

Instead of pre-allocating fixed capacity, these methods grow the network incrementally in response to new tasks. This can involve adding new neurons, layers, or entire modules. The expansion is often selective and data-driven, adding capacity only where needed to learn new features without disrupting old ones. Techniques like PackNet sequentially prune less important weights from a dense network to free up space, then retrain the freed parameters for the next task, effectively performing in-place parameter isolation over time.

Inspired by modular brain theories, these methods use a large, fixed overparameterized model where, for any given input, only a sparse subset of pathways is activated. Different tasks activate different, potentially overlapping, pathways. Mechanisms include:

• Mixture of Experts (MoE): A gating network routes inputs to a small selection of specialized expert sub-networks.
• Conditional Computation: Task-specific or input-specific binary masks determine which parts of the network are active. The goal is to achieve high capacity and flexibility while minimizing interference through sparsity.

Architectural methods are often contrasted with regularization-based and replay-based approaches.

• Regularization (e.g., EWC, SI): Adds penalty terms to the loss to slow down changes to important weights. Does not modify architecture.
• Replay (e.g., ER, GEM): Rehearses on stored/generated past data. Uses a standard, fixed architecture.
• Architectural: Dynamically modifies the model structure. It provides the strongest guarantee against interference but incurs structural complexity and potential parameter overhead. In practice, hybrid approaches (e.g., expansion + replay) are common to balance retention, efficiency, and forward transfer.

Architectural methods inherently increase model size, leading to significant memory and storage costs. Progressive Neural Networks and Parameter Isolation techniques like Hard Attention to the Task (HAT) create dedicated subnetworks or columns for each task.

• Static Growth: Model size grows linearly or sub-linearly with the number of tasks, which can become prohibitive for long task sequences.
• Inference Memory: The entire expanded architecture must be loaded into memory during inference, unlike regularization-based methods that use a single fixed-size model.
• Storage: Checkpoint sizes increase substantially, impacting model registry and deployment pipelines.

These methods excel at preventing negative backward transfer (catastrophic forgetting) but vary in their ability to facilitate positive forward transfer (improving new task learning via old knowledge).

• Progressive Networks: Enable strong forward transfer via lateral connections to frozen prior columns, but reuse is architecturally constrained.
• Parameter Isolation/Masking: (e.g., HAT) intentionally prevent parameter sharing to avoid interference, which can limit positive forward transfer unless explicitly designed via routing mechanisms.
• Trade-off: There is often a direct tension between absolute forgetting prevention (isolation) and efficient knowledge reuse (transfer).

Most architectural methods require an explicit task identifier during both training and inference to select the correct subnetwork, mask, or pathway. This introduces significant system complexity.

• Routing Logic: Production systems must maintain and provide a reliable task ID, which is non-trivial in task-free or blurry-boundary scenarios.
• Error Propagation: Incorrect task ID at inference leads to completely wrong model behavior, as a different task's dedicated parameters are activated.
• Soft vs. Hard Routing: Learned routing controllers add training complexity but can mitigate the need for explicit IDs.

The training dynamics are more complex than standard end-to-end backpropagation, impacting development time and computational cost.

• Phased Training: Methods like Progressive Networks require sequential, non-parallelizable training of new columns.
• Mask Optimization: Learning binary masks (as in HAT) involves straight-through estimators or other non-standard gradient techniques.
• Search Costs: Methods that dynamically expand architecture may involve Neural Architecture Search (NAS) components, drastically increasing meta-training costs.

Despite having dedicated parameters, inference does not always require executing the entire expanded model. However, optimization is challenging.

• Conditional Execution: Only the activated subnetwork (e.g., a specific column or masked pathway) needs to be executed, potentially keeping latency similar to a base model.
• Routing Overhead: The logic to select the correct subnetwork adds latency. For HAT, applying binary masks is computationally cheap.
• Hardware Utilization: Sparse, conditional execution can lead to underutilization of parallel hardware (GPUs/TPUs) designed for dense computation.

Implementing architectural methods on large, pre-trained models (e.g., Transformers) and within standard ML frameworks presents engineering hurdles.

• Transformer Adaptation: Isolating parameters in attention blocks or feed-forward networks requires careful layer-specific design.
• Framework Support: Custom masking, gradient stopping, and conditional forwarding often require low-level framework modifications or bypassing standard nn.Module patterns.
• Compatibility: Difficult to combine with other efficiency techniques like quantization or pruning, which operate on the global parameter set.

An architectural method where a new, task-specific neural network column is instantiated for each new task. Lateral connections from all previous columns allow feature reuse, while old parameters are frozen to prevent forgetting. This approach provides strong isolation but leads to linear parameter growth with each task, which can be computationally expensive for long task sequences.

A family of strategies that allocate non-overlapping subsets of a model's parameters to different tasks, completely avoiding interference. Key techniques include:

• Hard Attention to the Task (HAT): Learns binary masks over network units for selective activation.
• PackNet: Iteratively prunes and freezes weights for old tasks, freeing capacity for new ones.
• SupSup (Supermasks in Superposition): Stores multiple tasks within a single network by learning unique binary masks for each. These methods offer strong forgetting prevention but require task identity at inference.

Model designs that can expand, route, or sparsely activate components to accommodate new data streams. Unlike static networks, these architectures adapt their structure, which can include:

• Neural Architecture Search (NAS) for continual learning.
• Dynamic routing networks that activate different sub-networks per input.
• Expanding layers or neurons to add capacity for novel patterns. This approach balances the stability-plasticity dilemma by adding plasticity where needed while preserving stable core components.

A specific parameter isolation method that learns task-specific, hard binary attention masks over the units (e.g., neurons, attention heads) in a neural network. During training for a task, a sigmoid-based gating mechanism is used, which is later thresholded to a binary mask for inference. This ensures zero gradient flow to parameters assigned to other tasks, providing strict isolation. A key challenge is managing capacity as the fixed network is partitioned among an increasing number of tasks.

A core distinction in continual learning strategy. Architectural methods (like Progressive Nets or HAT) modify the network structure to isolate knowledge. Regularization methods (like EWC or SI) add penalty terms to the loss function to constrain weight changes. Architectural approaches often prevent forgetting more rigorously but can be less parameter-efficient and may require task identity. Regularization methods are more parameter-efficient but may struggle with severe distribution shifts or long task sequences.

The fundamental engineering challenge of architectural methods. Progressive expansion prevents forgetting but causes linear parameter growth. Fixed-capacity isolation (e.g., HAT) requires intelligent partitioning of a finite resource. Key metrics for evaluation include:

• Final model size relative to number of tasks.
• Inference compute (FLOPs) per task.
• Forward transfer: Ability to improve on future tasks.
• Backward transfer: Impact on past task performance. Optimal strategies depend on whether the system prioritizes performance, efficiency, or a known task horizon.