Elastic Weight Consolidation (EWC)

EWC identifies important parameters by computing the Fisher Information Matrix on the original task's data. This matrix approximates the curvature of the loss function, revealing which weights are most sensitive to perturbation. Parameters with high Fisher values are deemed critical for preserving prior knowledge. The diagonal approximation is typically used for computational efficiency, avoiding the full quadratic cost of the matrix.

The core mechanism adds a quadratic penalty to the loss function during new task training:

• L(θ) = L_new(θ) + Σ (λ/2) * F_i * (θ_i - θ*_i)^2
• L_new is the loss for the new domain (e.g., legal text)
• F_i is the Fisher information for parameter i
• θ*_i is the optimal parameter value from the previous task
• λ sets the overall importance of remembering old tasks This acts as a spring anchoring each parameter to its prior optimum, with stiffness proportional to its importance.

EWC is directly inspired by the neuroscience of memory consolidation. The mammalian neocortex exhibits 'protected' synapses that are resistant to overwriting, providing a mechanistic model for how brains learn new skills without erasing old ones. EWC replicates this by rendering high-Fisher parameters less plastic, mimicking the reduced synaptic plasticity observed in consolidated neural circuits.

When adapting a general-purpose model to legal text via Domain-Adaptive Pre-Training (DAPT), EWC prevents the model from forgetting its foundational language understanding:

• The Fisher matrix is computed on a representative sample of general-domain text
• During continued training on legal corpora, parameters crucial for general syntax and semantics are constrained
• The model acquires legal terminology and reasoning without suffering a collapse in general language performance This is critical for maintaining capabilities like summarization and question-answering that transfer to the legal domain.

Unlike Experience Replay, which requires storing and retraining on original data, EWC imposes no data storage overhead. This is advantageous when:

• Original training data is proprietary or cannot be retained due to privacy constraints
• Storage infrastructure for a replay buffer is unavailable
• The computational cost of interleaving old data is prohibitive However, EWC's quadratic penalty is a local approximation and may be less effective than true rehearsal for tasks with highly non-linear loss landscapes.

EWC scales to sequential learning across multiple domains by maintaining a running Fisher estimate. After training on task B, the Fisher matrix is updated to reflect the combined importance of tasks A and B. This creates a cumulative constraint that protects all prior knowledge. The technique is foundational for building models that incrementally learn across diverse legal sub-domains—contracts, statutes, case law—without catastrophic interference.

The mathematical backbone of EWC. The Fisher Information Matrix quantifies how sensitive the model's output is to changes in each parameter. Parameters with high Fisher values are deemed critical to prior knowledge and are heavily penalized during new task training. In legal domain adaptation, this matrix identifies which weights encode fundamental language understanding versus those free to specialize in statutory interpretation.

The broader training paradigm that EWC serves. DAPT continues pre-training a foundation model on a large, unlabeled legal corpus to adapt its internal representations. Without EWC or similar regularization, DAPT risks catastrophic forgetting of general knowledge. EWC acts as a guardrail, ensuring the model gains legal expertise without sacrificing its foundational language understanding.