Catastrophic Forgetting

Catastrophic forgetting is the manifestation of the stability-plasticity dilemma in neural networks. A model must be plastic enough to learn new legal terminology and reasoning patterns from a domain-specific corpus, yet stable enough to retain the general syntax, semantics, and world knowledge acquired during its original pre-training. Without intervention, gradient updates from a narrow legal data distribution will overwrite weights critical for general language understanding, causing the model to abruptly lose capabilities like common-sense reasoning or translation.

A foundational regularization technique that identifies parameters most critical to a model's previously learned tasks. EWC calculates the Fisher Information Matrix to estimate the importance of each weight to the original general-domain performance. During legal fine-tuning, it applies a quadratic penalty that constrains changes to high-importance weights, effectively 'freezing' the model's core linguistic knowledge while allowing less critical parameters to adapt to legal text. This is mathematically analogous to a spring anchoring each parameter to its old value.

A continual learning strategy that mitigates forgetting by interleaving data from the original general-domain training distribution with new legal data during fine-tuning. Key mechanisms include:

• Replay Buffer: A memory bank storing a representative sample of general text (e.g., Wikipedia, books) that is periodically replayed.
• Generative Replay: Instead of storing raw data, a separate generative model is trained to produce pseudo-samples of the original distribution, avoiding privacy and storage concerns.
• Maximally Interfered Retrieval (MIR): An advanced sampling strategy that selects replay samples most likely to be forgotten, maximizing the efficiency of the replay buffer.

An architectural approach that sidesteps forgetting entirely by freezing the original model and training a new, separate sub-network for the legal domain. Each new task instantiates a fresh column of layers that receives lateral connections from all previously frozen columns. This guarantees zero interference with prior knowledge but introduces linear parameter growth with each new domain, making it computationally expensive for a broad range of legal sub-specialties. It is most effective when absolute preservation of a base model's capabilities is non-negotiable.

An unsupervised regularization method that tracks the contribution of each synapse to the reduction of the global loss function over the course of a learning trajectory. Unlike EWC, SI computes importance weights online during training without requiring explicit task boundaries. When transitioning to legal domain adaptation, SI applies a per-synapse consolidation penalty proportional to its accumulated importance, allowing the model to autonomously protect parameters that were crucial for its historical performance on general language tasks.

A technique that uses the original generalist model as a teacher to guide the legal domain student model. During fine-tuning, the student is trained not only on the legal corpus but also to match the output logits of the frozen teacher on general-domain prompts. This distillation loss acts as a soft constraint, preserving the teacher's internal decision boundaries and output distribution. The method is particularly effective for preserving nuanced generative behaviors, such as stylistic fluency and refusal boundaries, that are difficult to capture with weight-based regularization alone.

A continual learning strategy that mitigates forgetting by interleaving data from the original general-domain pre-training corpus with new legal data. By periodically 'replaying' general text, the model maintains its broad language understanding. This approach is particularly effective for causal language models but requires storing or regenerating samples from the original massive dataset, introducing significant storage and computational overhead.

An intrinsic evaluation metric that quantifies how 'surprised' a model is by a held-out legal text. A decreasing perplexity score on a legal validation set indicates successful domain adaptation. However, tracking perplexity on a general-domain corpus simultaneously is critical; a sudden spike in general perplexity is the primary quantitative signal that catastrophic forgetting is occurring, even as legal performance appears to improve.

A compression technique that can also serve as a forgetting mitigation strategy. A 'student' model is trained to replicate the output distribution of a frozen, generalist 'teacher' model while also learning from new legal data. This dual objective preserves general capabilities without requiring access to the original pre-training data. The student learns to mimic the teacher's behavior on general prompts while specializing in legal reasoning.

The strategic composition of the continued pre-training corpus is a first-line defense against catastrophic forgetting. A well-designed mix includes not just narrow legal texts but also general-domain samples. Data stratification ensures proportional representation of diverse sources, preventing the model from overfitting to a single legal style and abruptly discarding its foundational language understanding.