Citation Masking

By replacing a specific citation like 'Brown v. Board of Education, 347 U.S. 483 (1954)' with a generic [CASE_CITATION] token, the model is forced to learn the pragmatic function of the reference—that it provides binding precedent for a constitutional claim—rather than memorizing a static string. This builds a deeper, more transferable understanding of legal reasoning structures.

A primary driver of legal hallucination is the model's memorization of surface-level citation patterns without grounding in factual authority. Citation masking directly combats this by severing the brittle link between a rhetorical context and a specific case name. The model learns to reason about why authority is needed, reducing the generation of syntactically plausible but entirely fabricated citations like 'Smith v. Air Canada'.

Implementation involves a deterministic pre-processing step using a legal tokenizer trained on domain corpora. The process identifies citation patterns via regular expressions or a named entity recognition model and replaces them with a small set of reserved tokens:

• [CASE_CITATION] for judicial opinions
• [STATUTE_CITATION] for legislative codes
• [REGULATION_CITATION] for administrative rules This expands the model's vocabulary with functional tokens.

A model trained on unmasked U.S. federal citations will fail catastrophically when asked to reason about U.K. or Canadian law because it has overfit to specific reporter formats. Citation masking creates a jurisdiction-agnostic reasoning skeleton. The model learns that a [CASE_CITATION] following a proposition serves a universal argumentative function, enabling zero-shot transfer to unseen legal systems.

Citation masking is a natural extension of the standard Masked Language Modeling (MLM) objective used in encoder models like Legal-BERT. Instead of randomly masking 15% of tokens, a targeted masking strategy is applied to citation spans. The model is then tasked with predicting not the original citation string, but the functional category of the masked token, reinforcing the semantic role of authority in legal discourse.

When a model memorizes specific case citations during pre-training, it can artificially inflate performance on downstream tasks like case outcome prediction. If the test set contains a case the model has memorized, it can 'cheat' by recalling the outcome rather than reasoning from facts. Citation masking eliminates this benchmark leakage vector, ensuring evaluation metrics reflect genuine legal reasoning capability, not a memorized lookup table.

The process of continuing to train a foundation model on a large, unlabeled legal corpus. Citation masking is a critical pre-processing step within DAPT, forcing the model to learn the contextual function of authority rather than memorizing specific case strings. This prevents the model from relying on surface-level citation patterns and instead builds a deeper understanding of how legal arguments are supported.

A text tokenization model trained on legal corpora to optimize subword splitting for domain-specific vocabulary. A specialized tokenizer is essential for citation masking to work effectively:

• It must correctly identify the boundaries of a citation (e.g., 347 U.S. 483) as a distinct unit to be masked
• It reduces the out-of-vocabulary rate for terms like res judicata or statutory references
• Standard tokenizers often fragment citations into meaningless subwords, breaking the masking logic

The process of identifying and removing near-duplicate legal documents from a training corpus. This is a prerequisite for effective citation masking because:

• Duplicate cases create spurious memorization pathways that undermine the purpose of masking
• A model can cheat by memorizing the same citation string across multiple copies, even if masked in one instance
• Proper de-duplication ensures the model is forced to learn the reasoning function of a citation, not its surface form

A safety metric quantifying how often a model generates syntactically plausible but factually fabricated citations. Citation masking directly targets this metric during pre-training. By replacing citations with special tokens like [CITE], the model learns to predict that an authority is needed without hallucinating a specific, incorrect case name. Post-training, a retrieval module fills the masked slot with a verified citation from a ground-truth database.

A critical failure where evaluation data is inadvertently included in the pre-training corpus. Citation masking provides a structural defense against a specific form of leakage:

• If a model memorizes Smith v. Jones, 123 F.3d 456 from training data, it can cheat on downstream tasks
• Masking all citations during pre-training prevents the model from storing exact citation strings
• This ensures that benchmark performance reflects genuine legal reasoning, not memorized case-to-outcome mappings

A self-supervised approach that pulls semantically similar legal text pairs together in the embedding space. Citation masking complements this technique:

• A masked citation like [CITE] becomes a semantic anchor—two paragraphs citing the same authority for the same proposition will have similar representations around the mask token
• This builds a latent understanding of precedential relationships without explicit citation graph supervision
• Frameworks like SimCSE can leverage these masked structures for improved legal retrieval