Faithfulness Metric

The core mechanism of any faithfulness metric is measuring factual consistency—the degree to which every atomic claim in a generated text is logically entailed by the source document. This is typically operationalized through Natural Language Inference (NLI) models that classify each generated statement as entailed, contradicted, or neutral relative to the source. A perfectly faithful output has zero contradictions and zero unsupported (neutral) claims.

• Entailment: The source text logically implies the generated statement
• Contradiction: The generated statement directly conflicts with the source
• Neutral: The statement introduces information not present in the source

Before scoring, a faithfulness metric decomposes both the source document and the generated text into atomic claims—minimal, self-contained factual assertions that can be independently verified. For example, the sentence 'The court ruled on Tuesday that the patent was invalid' decomposes into:

• The court issued a ruling
• The ruling occurred on Tuesday
• The ruling concerned a patent
• The patent was found invalid

This decomposition enables fine-grained verification, preventing a model from receiving a high score by balancing a major fabrication with several trivial truths.

A critical sub-component of faithfulness metrics is contradiction detection—the computational task of identifying mutually exclusive statements between the source and the generated output. This is particularly vital in legal AI, where a hallucinated contradiction can invert a legal conclusion. Advanced metrics use span-level alignment to pinpoint the exact text segments in conflict.

• Direct contradiction: Source says 'liable,' output says 'not liable'
• Numerical contradiction: Source says '$500,000,' output says '$50,000'
• Temporal contradiction: Source says 'effective June 1,' output says 'effective July 1'

Faithfulness metrics distinguish between two categories of hallucination:

Extrinsic Hallucination: The model introduces information entirely absent from the source material. This is the classic 'fabrication' problem where a model invents a non-existent case citation or contractual clause.

Intrinsic Hallucination: The model misrepresents or distorts information that is present in the source. For example, it correctly identifies a party name but attributes the wrong legal obligation to them.

A robust metric must detect both types, as intrinsic hallucinations are often more dangerous because they appear superficially grounded.

Modern faithfulness metrics employ alignment-based evaluation rather than simple overlap measures. Instead of calculating ROUGE or BLEU scores against a reference, these metrics create a direct mapping between each generated claim and the specific source segment that supports or contradicts it. This produces an attribution graph that serves as a transparent audit trail.

• Claim-to-source mapping: Every output claim linked to a source passage
• Support weight: The strength of the entailment relationship
• Coverage score: The proportion of output claims with valid source grounding

A classification task that determines whether a hypothesis can be logically inferred from a premise. In faithfulness evaluation, the generated summary serves as the hypothesis and the source document as the premise. An NLI model classifies each claim as entailed, contradicted, or neutral, providing a fine-grained factual consistency score.

• Entailment: The claim is fully supported by the source
• Contradiction: The claim directly conflicts with the source
• Neutral: The claim introduces information not addressed in the source

Modern faithfulness metrics often use NLI as their underlying computational engine, decomposing long texts into atomic claims before running pairwise entailment checks.

The automated process of verifying that every factual claim in a generated text is explicitly supported by the provided source document. Unlike general faithfulness metrics that produce aggregate scores, groundedness detection operates at the claim level, flagging specific spans of unsupported text.

• Identifies extrinsic hallucinations: facts invented beyond the source
• Detects intrinsic hallucinations: distortions of source material
• Enables span-level highlighting for human review workflows

In legal AI, groundedness detection serves as a critical guardrail, preventing fabricated case citations and misrepresented statutory language from reaching end users.

A metric that quantifies the degree to which a generated statement can be directly linked to a specific segment of a source document. While faithfulness measures overall consistency, attribution scoring provides provenance granularity, ensuring every legal conclusion has a verifiable origin.

• Direct Attribution: The claim maps to a single, identifiable source passage
• Partial Attribution: The claim synthesizes multiple source segments
• Unattributed: The claim has no identifiable source anchor

High attribution scores are essential for legal applications where citation integrity is non-negotiable and every assertion must withstand adversarial scrutiny.

The computational task of identifying mutually exclusive statements within a single document or across a multi-document corpus. In legal AI, contradiction detection surfaces logical inconsistencies that undermine the coherence of generated analysis.

• Intra-document contradictions: Conflicting claims within one generated summary
• Cross-document contradictions: Inconsistencies when synthesizing multiple sources
• Temporal contradictions: Statements that conflict due to chronological ordering

This capability is critical for multi-document legal reasoning, where a model must reconcile holdings from different jurisdictions or identify when a later statute supersedes earlier precedent.

A paired set of metrics specifically designed for legal AI evaluation. Citation Recall measures the proportion of factual claims that are correctly supported by a citation, while Citation Precision measures the proportion of provided citations that genuinely support their associated claim.

• High Recall, Low Precision: The model cites frequently but often to irrelevant sources
• Low Recall, High Precision: The model cites sparingly but accurately
• F1 Score: The harmonic mean balancing both concerns

These metrics directly address the legal profession's requirement for verifiable authority, detecting both unsupported assertions and fabricated references that would constitute professional malpractice.

A faithfulness sub-metric that evaluates whether a model's response is strictly derived from the user-provided context, penalizing the introduction of external knowledge or assumptions not present in the input. This is distinct from general faithfulness, which may consider the model's training data as an acceptable knowledge source.

• Strict adherence: Only information from the provided context window
• Parametric contamination: The model injects memorized training data
• Instruction following: The model obeys explicit constraints in the prompt

For legal applications, context adherence ensures that a model analyzing a specific contract does not import clauses from similar agreements seen during training, maintaining document-level fidelity.