Normative Faithfulness Metric

Measures the proportion of generated deontic operators (obligations, permissions, prohibitions) that are actually present in the source text.

• Formula: True Positives / (True Positives + False Positives)
• Key Insight: A high score indicates the model is not inventing duties that do not exist
• Example: If a model generates 10 obligations but only 7 are grounded in the contract, precision is 0.7
• Penalizes: Hallucinated obligations and fabricated prohibitions

Measures the proportion of deontic operators in the source text that are successfully captured in the generated output.

• Formula: True Positives / (True Positives + False Negatives)
• Key Insight: A high score indicates the model is not omitting critical duties
• Example: If a contract contains 12 obligations but the summary only reflects 9, recall is 0.75
• Penalizes: Omission of normative content, which creates compliance risk

Evaluates whether the generated text preserves the correct jural correlatives defined by Hohfeldian analysis.

• Right/Duty pairs: Does every generated right have a corresponding duty bearer?
• Power/Liability pairs: Are generated powers matched with the correct liability?
• Privilege/No-right pairs: Are permissions correctly structured?
• Example: A clause stating 'Party A may audit' (privilege) must not be transformed into 'Party B must submit to audit' (duty) unless the source explicitly establishes both correlatives

Assesses whether the generated text correctly handles conditional obligations that arise when primary duties are violated.

• Primary obligation: The ideal normative state
• CTD obligation: The fallback rule triggered by non-compliance
• Evaluation: Does the output preserve the logical relationship between primary and secondary duties?
• Example: 'The contractor shall deliver by June 1. If delivery is late, the contractor shall pay $500 per day.' The metric checks that the penalty obligation is correctly conditioned on the violation of the primary duty

Measures the model's ability to correctly resolve conflicting norms using established legal principles.

• Lex Superior: Does the model defer to constitutional over statutory norms?
• Lex Specialis: Does the specific provision override the general one?
• Lex Posterior: Does the later-enacted rule prevail?
• Example: If a federal statute prohibits data sharing but a state regulation permits it, the metric evaluates whether the output correctly applies the supremacy principle

Evaluates whether the generated text accurately represents the temporal states of obligations: activation, fulfillment, violation, and expiration.

• Activation: Is the triggering condition correctly identified?
• Fulfillment: Is the satisfaction condition accurately described?
• Violation: Is the breach state properly flagged?
• Expiration: Is the termination condition preserved?
• Example: 'The NDA obligations survive for 5 years post-termination.' The metric verifies that the survival period is not truncated or extended in the generated summary

The structured labeling framework used to create gold-standard training data for normative NLP models. This schema tags legal text with deontic categories—obligation, permission, prohibition—and their attributes (bearer, counterparty, triggering condition). Without a rigorous annotation schema, calculating a Normative Faithfulness Metric is impossible, as there is no ground truth against which to compare a model's generated deontic content.

• Defines the taxonomy of normative concepts
• Enables inter-annotator agreement measurement
• Forms the evaluation bedrock for deontic extraction

A natural language inference task that determines whether a textual premise normatively entails an obligation, permission, or prohibition in a conclusion. This is the direct NLP task underlying the Normative Faithfulness Metric. A generated summary is faithful if, and only if, its deontic content is entailed by the source document.

• Benchmarks deontic reasoning capability
• Provides a binary or graded faithfulness signal
• Used to audit hallucinated obligations

The suite of techniques for preventing factual fabrication in legal generative models. The Normative Faithfulness Metric is the primary quantitative KPI for these mitigation strategies. A hallucinated obligation—an invented duty not present in the source statute or contract—is the most dangerous failure mode in legal AI.

• Retrieval-Augmented Generation (RAG) grounds outputs in source text
• Constrained decoding restricts the model's output vocabulary
• Self-consistency checks sample multiple reasoning paths

A runtime constraint mechanism that filters or validates the output of a generative AI model to ensure it does not prescribe illegal actions or violate encoded norms. The Normative Faithfulness Metric is used to calibrate the sensitivity of these guardrails.

• Acts as a safety layer between model output and user
• Can block, rewrite, or flag non-faithful generations
• Uses the faithfulness score as a threshold trigger

The algorithmic detection and reconciliation of contradictory legal rules. A faithful generation must not only capture individual norms but also correctly represent their resolution state. The Normative Faithfulness Metric penalizes models that output unresolved or incorrectly resolved conflicts.

• Applies principles like lex superior and lex specialis
• Requires modeling of normative hierarchies
• Critical for multi-jurisdictional analysis

An OASIS standard XML-based markup language for encoding legal rules with deontic semantics. It provides a formal, machine-readable interchange format for normative knowledge. A Normative Faithfulness Metric can be computed deterministically by comparing a generated LegalRuleML representation against a ground-truth LegalRuleML encoding of the source.

• Enables deterministic evaluation of faithfulness
• Supports obligation, permission, and prohibition operators
• Facilitates interoperability between reasoning engines