Masked Language Modeling (MLM)

MLM randomly masks a percentage of input tokens (typically 15%) and forces the model to predict the original vocabulary ID of the hidden word. Unlike Causal Language Modeling (CLM) , which only sees left context, MLM allows the model to attend to tokens both before and after the mask. This bidirectional attention is critical for understanding complex legal syntax where the meaning of a term like 'consideration' depends on its full contractual context.

The core architectural advantage of MLM is its ability to fuse left and right context into a single representation. For a masked statute like 'The [MASK] shall have the authority to adjudicate,' the model simultaneously weighs 'The' and 'shall have the authority' to predict 'court' or 'tribunal.' This deep contextualization enables Domain-Adaptive Pre-Training (DAPT) to produce superior legal embeddings for downstream tasks like Legal Text Summarization and Citation Network Analysis.

Standard random masking is often suboptimal for legal corpora. Advanced strategies include:

• Whole Word Masking: Masking entire words like 'jurisdiction' instead of subword pieces, preventing the model from cheating via partial tokens.
• Citation Masking: Replacing legal citations (e.g., '347 U.S. 483') with a special [CITE] token, forcing the model to learn the rhetorical function of authority rather than memorizing case strings.
• Span Masking: Masking contiguous sequences of tokens to teach the model to reconstruct longer legal phrases and clauses.

MLM is the defining pre-training objective for encoder-only transformer architectures like BERT, RoBERTa, and Legal-BERT. These models are not designed for text generation but excel at natural language understanding. In the legal domain, an MLM-pretrained encoder produces contextualized embeddings that power Contract Clause Extraction, Legal Argument Mining, and semantic similarity systems. The resulting [CLS] token representation serves as a dense summary of an entire legal document.

A key efficiency consideration is that MLM only generates a training signal from the 15% of tokens that are masked. The remaining 85% of tokens are ignored in the loss calculation, making MLM less sample-efficient per epoch than Causal Language Modeling (CLM) . To compensate, legal models like CaseLaw-BERT are typically trained for more epochs on carefully stratified Legal Data Mix corpora. The trade-off is accepted because the resulting bidirectional representations are far more powerful for classification and extraction tasks.

The primary intrinsic metric for an MLM-trained legal model is Legal Perplexity on a held-out corpus of statutes and contracts. A lower perplexity score indicates the model has internalized the statistical patterns of legal language. For example, a model achieving a perplexity of 3.2 on the EDGAR contracts corpus has a much stronger grasp of transactional syntax than a general-domain model scoring 8.7. This metric directly correlates with downstream performance on tasks from the LexGLUE benchmark.

The autoregressive counterpart to MLM, where the model predicts the next token in a sequence based solely on preceding context. Unlike MLM's bidirectional attention, CLM uses a unidirectional (left-to-right) attention mask, making it the foundation for generative models like GPT-4. In legal AI, CLM excels at text generation tasks such as drafting clauses and summarization, while MLM-based encoders are preferred for understanding tasks like classification and entity extraction.

Algorithms like Byte-Pair Encoding (BPE) and SentencePiece that split text into frequent subword units, balancing vocabulary size against coverage of rare terms. For legal MLM pre-training, a domain-specific tokenizer is critical—it prevents splitting terms like 'res judicata' or '28 U.S.C. § 1332' into meaningless fragments. A well-trained legal tokenizer directly reduces the out-of-vocabulary rate and improves the model's ability to learn coherent representations of statutory language.

The process of continuing to train a foundation model on a large, unlabeled domain-specific corpus using the same MLM objective. For legal models, this means taking a BERT-base checkpoint and continuing pre-training on millions of case law documents and contracts. DAPT adapts the model's internal representations to legal syntax and semantics without requiring labeled data, dramatically improving downstream performance on tasks like contract clause extraction and case outcome prediction.

A specialized pre-processing strategy where legal citations are replaced with special tokens during MLM pre-training. Instead of predicting '347 U.S. 483', the model sees [CITE] and must learn the contextual function of the authority—why a case is cited—rather than memorizing specific citation strings. This technique prevents the model from hallucinating plausible but incorrect citations and encourages deeper reasoning about precedential relationships.

An intrinsic evaluation metric measuring how surprised a language model is by a held-out legal text. A lower perplexity indicates the model has better internalized the statistical patterns of legal language. For MLM-based models, perplexity is computed over the masked tokens only. While useful for monitoring pre-training progress, low perplexity does not guarantee good downstream performance—a model may perfectly predict legal syntax while failing at complex reasoning tasks.

An IO-aware exact attention algorithm that dramatically accelerates the self-attention mechanism central to MLM architectures. By optimizing memory reads and writes between GPU high-bandwidth memory and on-chip SRAM, FlashAttention reduces the quadratic memory complexity of attention. This makes it feasible to pre-train legal encoders on longer sequence lengths, enabling models to process entire multi-page contracts or judicial opinions in a single forward pass without truncation.