Catastrophic Forgetting

A synaptic consolidation technique that identifies parameters critical to previously learned tasks and penalizes their modification during new training. EWC computes the Fisher Information Matrix to estimate parameter importance, applying a quadratic penalty proportional to the diagonal of the Fisher. This constrains updates to less important weights, preserving old knowledge while allowing plasticity in underutilized parameters. In federated settings, the server can aggregate Fisher matrices alongside model weights to maintain a global importance map.

The training paradigm where a federated model learns sequentially from a non-stationary stream of client data over time. The primary goal is to integrate new knowledge from evolving distributions while explicitly mitigating catastrophic forgetting of previously learned global patterns. This requires balancing local plasticity on new client data with global stability on historical knowledge, often through rehearsal buffers or dynamic architecture expansion.

A seminal regularization technique that identifies and protects weights critical to previously learned tasks. EWC computes the Fisher information matrix to measure parameter importance, then penalizes large updates to these crucial weights during new task training. This slows learning on high-importance connections, forcing the network to accommodate new knowledge in underutilized parameters and preserving old task performance.

A memory-based approach that interleaves new data samples with a stored subset of examples from previous tasks during training. By periodically retraining on old data, the network is constantly reminded of past distributions, preventing the gradient updates for new tasks from overwriting established representations. Key challenges include managing memory capacity and selecting representative samples in privacy-sensitive environments.

An unsupervised regularization method that tracks the cumulative contribution of each synapse to the reduction of the global loss over a task's training trajectory. When a new task begins, SI penalizes changes to synapses that were highly important for previous tasks, proportional to their accumulated importance. Unlike EWC, it computes importance online during training without requiring explicit task boundaries or Fisher information.

The fundamental tension in neural network learning between the stability required to retain existing knowledge and the plasticity needed to integrate new information. Catastrophic forgetting represents a catastrophic failure of stability. A perfectly stable network cannot learn anything new, while a perfectly plastic network forgets everything instantly. All continual learning strategies are essentially resource allocation problems balancing these two constraints.

An architectural approach that instantiates a new neural network column for each new task while freezing previously trained columns. Lateral connections from prior columns to the new column enable transfer learning without modifying old weights, completely eliminating catastrophic forgetting by design. The trade-off is linear growth in model parameters with the number of tasks, making it impractical for long task sequences.