Causal Language Modeling (CLM)

The defining architectural feature of CLM is the causal mask applied to the self-attention mechanism. This mask prevents each token from attending to any token to its right in the input sequence. By enforcing a strict left-to-right information flow, the model is forced to predict the future based solely on the past, preserving the autoregressive property necessary for coherent text generation. This contrasts with bidirectional attention in Masked Language Modeling (MLM) used by encoder models like BERT, which can see the full context but cannot generate text natively.

CLM is trained to maximize the probability of the next token in a sequence given all preceding tokens. The training objective is to minimize the cross-entropy loss between the predicted probability distribution over the vocabulary and the actual next token. For a legal corpus, this means the model learns to predict the next word in a statute, the next clause in a contract, or the next logical step in a judicial opinion. This simple objective scales effectively with massive datasets and compute, enabling the emergence of complex reasoning patterns.

During training, CLM uses teacher forcing: the ground-truth previous token is always fed as input, regardless of the model's own prediction. This stabilizes training but creates a discrepancy with inference, where the model must condition on its own previously generated tokens. This exposure bias can lead to compounding errors in long legal generations, where an early mistake in a citation or clause structure derails subsequent output. Techniques like scheduled sampling attempt to bridge this gap.

The legal sequence length—the maximum number of tokens a CLM can process in one forward pass—is a critical architectural constraint. Early models had 2,048-token windows, insufficient for multi-page contracts. Modern architectures using FlashAttention and Rotary Position Embeddings (RoPE) extend this to 128k+ tokens, enabling a model to hold an entire merger agreement or a complex statutory framework in its attention window for coherent, cross-document reasoning.

CLM outputs a probability distribution over the entire vocabulary for the next token. The temperature hyperparameter controls the shape of this distribution before sampling. A temperature near 0 makes the model deterministic, always selecting the highest-probability token—useful for extracting exact statutory text. A higher temperature flattens the distribution, increasing diversity and creativity, which is valuable for generating alternative contract clauses but raises the risk of legal hallucination.

To balance fluency and factual accuracy in legal text, CLM generation relies on decoding strategies beyond simple greedy selection. Top-k sampling restricts the model to the k most likely next tokens, while nucleus sampling (top-p) selects from the smallest set of tokens whose cumulative probability exceeds a threshold p. These methods prevent the model from sampling low-probability tokens that would introduce factual errors or nonsensical legal jargon, maintaining the high citation integrity required for professional use.

The primary alternative pre-training objective to CLM, used in encoder models like BERT. MLM predicts randomly masked tokens by attending to bidirectional context—both left and right of the mask. This builds deep syntactic understanding ideal for legal classification and extraction tasks, but cannot generate text autoregressively. Key differences from CLM:

• Uses bidirectional rather than unidirectional attention
• Excels at understanding tasks, not generation
• Forms the basis for models like Legal-BERT

An intrinsic evaluation metric measuring how surprised a CLM is by a held-out legal text. Lower perplexity indicates the model has better internalized the statistical patterns of legal language. Calculated as the exponential of the cross-entropy loss over a token sequence. For legal CLMs, perplexity is typically measured on domain-specific corpora like:

• Federal appellate opinions
• Commercial contracts
• Regulatory filings A model with a legal perplexity of 8.2 assigns higher probability to correct legal tokens than one scoring 15.6.

Algorithms like Byte-Pair Encoding (BPE) and SentencePiece that split text into frequent subword units, balancing vocabulary size against the need to represent rare legal terminology. A legal-optimized tokenizer prevents splitting critical terms into meaningless fragments. For example, a general tokenizer might fragment 'res judicata' into ['res', 'jud', 'ic', 'ata'], while a legal tokenizer preserves it as a single meaningful token, dramatically improving CLM training efficiency and generation quality.

The tendency of a neural network to abruptly lose its general language capabilities when continually pre-trained on a narrow legal domain. A CLM fine-tuned exclusively on SEC filings may forget how to generate coherent non-legal prose. Mitigation techniques include:

• Elastic Weight Consolidation (EWC): penalizes changes to parameters important for prior knowledge
• Experience Replay: interleaves original general-domain data with new legal data during continued training
• Progressive Networks: freezes original model weights and learns lateral connections for new domain knowledge

The maximum number of tokens a CLM can process in a single forward pass, a critical architectural constraint for legal AI. Legal documents frequently exceed standard context windows:

• A typical commercial lease: 8,000-15,000 tokens
• A Supreme Court opinion: 10,000-25,000 tokens
• A merger agreement: 50,000+ tokens Techniques like FlashAttention and RoPE scaling enable efficient training on these long sequences, allowing the CLM to maintain causal coherence across entire multi-page documents.

A stable alignment algorithm that directly optimizes a CLM's policy from human preference data without training a separate reward model. For legal models, DPO fine-tunes generation behavior using paired examples of preferred (citations accurate, reasoning sound) and dispreferred (hallucinated, conclusory) outputs. This produces legal CLMs that generate citationally-accurate, well-reasoned text while avoiding the instability of reinforcement learning from human feedback (RLHF).