LegalBench

LegalBench organizes its evaluation tasks into a rigorous taxonomy based on the type of legal reasoning required, not the document format. This structure allows developers to pinpoint specific cognitive failures.

• Issue-Spotting: Identifying legal questions latent in a fact pattern
• Rule-Recall: Retrieving the correct black-letter law without fabrication
• Rule-Application: Mapping a known legal rule to a novel set of facts
• Conclusion-Formulation: Synthesizing analysis into a definitive legal judgment

This granularity prevents benchmark scores from masking critical weaknesses in a model's reasoning pipeline.

LegalBench is explicitly designed to penalize hallucination by treating incorrect legal assertions as strictly wrong, not partially correct. The primary metrics are:

• Exact Match Accuracy: The model's output must precisely match the legally correct answer; paraphrasing a wrong rule is scored as zero
• Abstention Rate: The benchmark measures how often a model refuses to answer rather than fabricate, rewarding calibrated uncertainty
• Citation Fidelity: For tasks requiring authority, the benchmark checks whether the cited source actually exists and supports the proposition

This zero-tolerance approach to fabrication makes LegalBench a critical tool for risk officers evaluating legal AI deployment.

The benchmark spans a wide range of American legal domains to test for transferable legal reasoning rather than narrow expertise. Tasks are drawn from:

• Contract Law: Interpreting clauses, identifying breaches, and calculating damages
• Constitutional Law: Applying established doctrinal tests to hypothetical scenarios
• Evidence: Determining admissibility under the Federal Rules of Evidence
• Criminal Procedure: Analyzing Fourth Amendment search and seizure questions
• Tort Law: Evaluating duty, breach, causation, and harm elements

This breadth ensures a model cannot achieve high scores by memorizing a single legal corpus.

LegalBench differs fundamentally from general-purpose benchmarks like MMLU or HELM in its treatment of legal truth as a binary, jurisdiction-bound concept.

• No Partial Credit: A model that states the wrong legal standard but writes a persuasive paragraph receives a score of zero
• Jurisdictional Grounding: Tasks specify the governing jurisdiction (e.g., U.S. federal law, California state law), and answers are judged against that specific authority
• Anti-Sycophancy Design: Tasks are structured to prevent models from agreeing with a legally incorrect prompt, testing for resistance to user error

This makes LegalBench a far more stringent test of factual reliability than benchmarks that reward fluent but incorrect text.

Two complementary metrics for evaluating the integrity of AI-generated legal analysis:

• Citation Recall: The proportion of factual claims in a generated text that are correctly supported by a citation. Low recall indicates unsupported assertions.
• Citation Precision: The proportion of provided citations that genuinely support the associated claim. Low precision signals fabricated references or irrelevant authority. Together, these metrics provide a rigorous framework for measuring a model's ability to provide verifiable provenance for every legal conclusion, directly addressing the core concern of hallucination in legal AI.

A quantitative evaluation framework that measures the factual consistency of a generated summary or answer relative to the source material. Unlike general quality metrics, faithfulness specifically identifies contradictions and unsupported fabrications. In the LegalBench ecosystem, faithfulness metrics are used to evaluate tasks like legal text summarization and multi-document reasoning, ensuring that a model's output does not introduce facts not present in the provided legal documents. This is a critical guardrail for any system performing contract analysis or case law synthesis.

The cognitive process of synthesizing information from multiple disparate source documents to derive a conclusion not explicitly stated in any single source. In legal AI, this is essential for tasks like cross-jurisdictional analysis and complex case synthesis. However, multi-hop reasoning is a primary source of hallucination because the model must:

• Correctly retrieve relevant facts from each document
• Maintain logical consistency across sources
• Avoid introducing external knowledge LegalBench includes tasks specifically designed to stress-test this capability, making it a key evaluation dimension for advanced legal reasoning systems.

A generation technique that forces a language model to output tokens that conform to a predefined formal grammar or JSON schema. This prevents structural hallucinations in machine-to-machine communication by ensuring the output is syntactically valid. In legal AI pipelines, schema-constrained decoding is used to guarantee that extracted contract clauses, case citations, and statutory references are formatted correctly for downstream processing. By restricting the model's output space to valid legal structures, this technique eliminates an entire class of formatting errors that could corrupt automated legal workflows.