Legal Model Card

The legal hallucination rate is a critical safety metric quantifying how often a model generates syntactically plausible but factually incorrect content, such as fabricated case citations or non-existent statutes. A comprehensive model card must disclose this rate, measured against a ground-truth legal database. Key evaluation methods include:

• Citation F1 Score: Balances the precision and recall of generated citations.
• Shepardizing: Automated verification of whether a cited case's holding is accurately represented.
• Statutory Existence Check: Validating generated statutory references against an official code.

Constitutional AI is an alignment method developed by Anthropic where a model is trained to self-critique and revise its outputs based on a predefined 'constitution' of principles. For a legal model, this constitution would encode rules like 'cite only real cases' and 'do not provide legal advice.' A model card should detail the alignment technique used, whether CAI, RLHF, or DPO, and the specific principles that govern the model's behavior. This transparency allows deployers to audit the model's normative guardrails.

Data stratification is a sampling technique ensuring a pre-training corpus proportionally represents key legal sub-domains, jurisdictions, and time periods. A model card's data composition section must detail this stratification to expose potential biases. Without it, a model may overfit to, for example, modern U.S. Supreme Court opinions and fail on historical English common law. The card should disclose the distribution of sources—statutes, contracts, case law, regulatory filings—and the temporal and geographic coverage of the training data.

Benchmark leakage is a critical failure where evaluation data is inadvertently included in the pre-training corpus, rendering performance metrics invalid. A rigorous model card must detail the de-contamination process used to detect and remove any overlap between training data and evaluation benchmarks like LexGLUE. This includes techniques such as n-gram overlap detection and exact substring matching. Full transparency on de-contamination is essential for establishing the trustworthiness of reported scores.

Direct Preference Optimization (DPO) is a stable alignment algorithm that directly optimizes a model's policy from human preference data, bypassing the need for a separate reward model. For legal models, DPO can be used to fine-tune outputs to be more helpful, harmless, and citationally accurate. A model card should specify the alignment method used and characterize the preference dataset, including the expertise of the human labelers (e.g., licensed attorneys vs. laypersons) and the specific criteria they used to rank model outputs.