Experience Replay

The core mechanism of Experience Replay involves mixing batches of new legal training data with replayed samples from the original general-domain corpus. During each training step, the model sees both a contract clause and a general text passage, forcing it to update its legal understanding without overwriting its foundational language skills. This interleaving ratio—typically between 1:1 and 1:10 (replay to new data)—is a critical hyperparameter that balances domain adaptation against general capability retention.

Without Experience Replay, a model undergoing domain-adaptive pre-training on legal texts will rapidly suffer from catastrophic forgetting—the abrupt loss of general language capabilities. The model's weight updates become biased toward the narrow legal distribution, causing it to fail on basic tasks like summarization or common-sense reasoning. Experience Replay acts as a regularization mechanism, maintaining the model's performance on general benchmarks like MMLU while improving its legal perplexity.

The replay buffer is a fixed-size memory store that holds representative samples from the original pre-training corpus. Key design decisions include:

• Buffer size: Typically 1-5% of the original corpus, stored in memory or on fast-access disk
• Sampling strategy: Uniform random sampling is standard, but importance-weighted sampling can prioritize diverse or rare linguistic patterns
• Refresh policy: Static buffers are common, but dynamic buffers can periodically resample to maintain distributional coverage

Experience Replay is often combined with Elastic Weight Consolidation (EWC) for stronger protection against forgetting. While replay provides explicit data reminders, EWC adds a quadratic penalty on parameter changes that are important for previous tasks. The Fisher Information Matrix identifies which weights are critical for general language performance, and the loss function penalizes their movement. This dual approach—data replay plus parameter regularization—provides state-of-the-art stability in legal domain adaptation.

The composition of the replay buffer must mirror the diversity of the original training distribution. For legal models, this means preserving samples from:

• General web text (Common Crawl, Wikipedia)
• Scientific and technical literature
• Multilingual corpora if the original model was multilingual
• Code repositories if code generation capabilities must be retained Failure to maintain this diversity results in a model that excels at contract analysis but cannot perform basic reasoning tasks outside the legal domain.

A subtle challenge in Experience Replay is gradient conflict—when the gradients from new legal data and replayed general data point in opposing directions. This can cause training instability and slower convergence. Advanced solutions include:

• Gradient surgery (PCGrad): Projecting conflicting gradients onto orthogonal planes before applying updates
• Multi-task optimization: Treating legal adaptation and general retention as separate objectives with dynamic loss weighting
• Alternating training: Switching between pure legal batches and pure replay batches on different steps

The strategic composition of a pre-training corpus from diverse legal sources—statutes, contracts, case law, and regulatory filings—to ensure a model develops broad reasoning capabilities. Experience Replay formalizes this by defining the exact ratio of replayed general data to new legal data in each training batch.

• Replay Ratio: A critical hyperparameter, often set between 1:1 and 10:1 (new data to replayed data).
• Buffer Composition: The selection of which general samples to retain in the replay buffer.

An intrinsic evaluation metric measuring how surprised a language model is by a held-out legal text. A lower perplexity score indicates a better internalized model of legal language patterns. It is used to monitor for catastrophic forgetting by tracking perplexity on a held-out general domain set.

• General Perplexity Spike: A sudden increase signals that forgetting is occurring.
• Validation: Must be paired with extrinsic task evaluation, as perplexity doesn't capture factual accuracy.

A training strategy that presents examples in a meaningful order, such as moving from short legal definitions to complex multi-document reasoning tasks. Experience Replay can be integrated into a curriculum by adjusting the difficulty of replayed samples over time.

• Legal Curriculum: Start with simple statutory text, progress to intricate multi-jurisdictional opinions.
• Replay Scheduling: Gradually reduce the frequency of replay as the model stabilizes on the new domain.