Domain-Adaptive Pre-Training (DAPT)

DAPT is not training from scratch. It begins with a powerful foundation model that already understands general language syntax and semantics. This model is then continually pre-trained on a domain-specific corpus—in this case, hundreds of billions of tokens from legal texts. This process updates the model's internal weights, shifting its statistical understanding of language from the general domain to the legal domain without requiring any labeled data. The objective remains the same (e.g., Causal Language Modeling), but the data distribution changes entirely.

The quality of DAPT is entirely dependent on the Legal Data Mix. A robust corpus is not a random collection of documents. It requires a strategic composition of diverse legal sources to build broad reasoning capabilities:

• Primary Law: Statutes, constitutions, and administrative codes.
• Case Law: Judicial opinions from multiple jurisdictions and court levels.
• Secondary Sources: Legal treatises, law review articles, and restatements.
• Transactional Documents: Contracts, SEC filings, and patent applications. Data Stratification ensures proportional representation across sub-domains and time periods to prevent overfitting.

A central risk of DAPT is Catastrophic Forgetting, where the model abruptly loses its general language capabilities as it specializes. To retain foundational knowledge while learning law, techniques like Elastic Weight Consolidation (EWC) are employed. EWC identifies the neural network parameters most critical for the original generalist task and penalizes significant changes to them. Experience Replay is another key method, which interleaves a small percentage of general-domain data with the new legal data during training to constantly refresh the model's broad understanding.

Raw legal text contains artifacts that can degrade model performance. Critical pre-processing steps are required:

• Citation Masking: Legal citations (e.g., '410 U.S. 113') are replaced with a special [CITE] token. This forces the model to learn the rhetorical function of a citation rather than memorizing specific case strings, improving generalization.
• Case Law De-duplication: Near-duplicate opinions (e.g., from a circuit court and its affirming Supreme Court case) are identified and removed to prevent Benchmark Leakage and ensure evaluation metrics reflect genuine reasoning, not memorization.

Pre-training on long legal documents demands significant compute. FlashAttention is an IO-aware algorithm that dramatically speeds up the self-attention mechanism and reduces its memory footprint, making it feasible to process lengthy contracts. Mixed-Precision Training uses BFloat16 for most operations to reduce memory usage, while ZeRO Optimization partitions model states across multiple GPUs, enabling the training of massive models that would otherwise exceed hardware limits. These techniques are not optional; they are prerequisites for cost-effective legal DAPT.

The success of DAPT is measured by intrinsic and extrinsic metrics. Legal Perplexity measures how 'surprised' the model is by a held-out legal text; a lower score indicates a better internalized model of legal language. Downstream, the Legal Hallucination Rate and Citation F1 Score are critical safety metrics that quantify the model's ability to generate factually correct, well-supported legal text. A Legal Model Card should transparently document all of these metrics, along with the training data composition and known limitations, for responsible deployment.