Conformal Prediction

Unlike asymptotic methods that require large sample sizes, conformal prediction provides a marginal coverage guarantee that holds for any finite dataset. If you specify a 90% confidence level, the true label will fall within the prediction set at least 90% of the time, regardless of the underlying data distribution. This is formalized as:

P(Y_test ∈ C(X_test)) ≥ 1 - α

• Exchangeability assumption: Requires only that calibration and test data are exchangeable, not i.i.d.
• No distributional assumptions: Works with any underlying data distribution
• Sample efficiency: Valid guarantees even with small calibration sets common in specialized legal domains

Conformal prediction operates as a post-hoc calibration layer that wraps around any pre-trained classifier or regressor without requiring access to model internals. This black-box compatibility makes it ideal for legal AI systems built on proprietary or third-party models.

• Classifier-agnostic: Works with logistic regression, random forests, gradient boosting, or neural networks
• No retraining required: Calibrates using a held-out calibration set without modifying model weights
• API-compatible: Can wrap models served behind REST endpoints, requiring only prediction scores
• Non-invasive: Preserves the original model's architecture and inference pipeline

Conformal prediction produces instance-adaptive prediction sets that reflect the model's uncertainty for each specific input. When the model is confident, the set is small (often a singleton); when uncertain, the set expands proportionally.

• Hard cases yield larger sets: A novel legal argument may produce a set containing multiple possible classifications
• Easy cases yield singletons: Routine contract clauses produce precise, single-label predictions
• Set size as uncertainty metric: The cardinality of the prediction set serves as an interpretable measure of model confidence
• Abstention mechanism: An empty prediction set signals that the input is out-of-distribution, triggering human review

The split-conformal or inductive variant divides available labeled data into a proper training set and a calibration set, avoiding the computational expense of full conformal prediction which requires retraining for every candidate label.

• Single model training: Train once on the proper training set, calibrate once on the calibration set
• Computationally efficient: Prediction set construction requires only sorting calibration scores
• Nonconformity scores: Measures how unusual a candidate label is relative to calibration examples
• Legal application: Enables real-time uncertainty quantification in contract review pipelines without latency penalties

Standard conformal prediction guarantees marginal coverage (averaged across all inputs), but legal applications often require coverage conditional on specific subpopulations. Mondrian conformal prediction and related extensions provide class-conditional guarantees.

• Mondrian conformal prediction: Ensures coverage within each class label, preventing systematic under-coverage for minority classes
• Group-conditional validity: Guarantees coverage across protected categories for fairness compliance
• Adaptive conformal inference: Dynamically adjusts thresholds for time-series regulatory data with distribution shift
• Legal fairness: Ensures that error rate guarantees hold equally for all case types, jurisdictions, or party demographics

The effectiveness of conformal prediction depends critically on the nonconformity measure—a scoring function that quantifies how unusual a candidate label is for a given input. Domain-specific measures dramatically improve set efficiency.

• Adaptive Prediction Sets (APS): Uses cumulative probability until the true class is included, producing smaller sets for well-calibrated classifiers
• Regularized Adaptive Prediction Sets (RAPS): Adds a penalty for set size, optimizing the efficiency-coverage trade-off
• Legal-specific measures: Can incorporate citation authority scores, temporal relevance, or jurisdictional proximity
• Efficiency metric: The average prediction set size—smaller sets with valid coverage indicate better nonconformity measures

A set of statistical techniques that enable a model to estimate the confidence of its own predictions. Unlike simple softmax probabilities, true uncertainty quantification distinguishes between aleatoric uncertainty (inherent data noise) and epistemic uncertainty (model ignorance).

• Bayesian neural networks model weight distributions rather than point estimates
• Monte Carlo dropout approximates Bayesian inference during inference
• Deep ensembles train multiple models to capture predictive variance

In legal AI, uncertainty quantification allows a system to flag high-risk classifications—such as a novel contract clause type—for mandatory human review rather than silently failing.

The discrepancy between a model's predicted confidence score and its actual empirical accuracy. A perfectly calibrated model's 90% confidence predictions should be correct exactly 90% of the time.

• Expected Calibration Error (ECE) bins predictions and measures the weighted average of confidence-accuracy gaps
• Reliability diagrams visually plot confidence against observed accuracy
• Temperature scaling is a post-hoc method that adjusts logit sharpness to improve calibration

Modern neural networks are notoriously overconfident, reporting 99% confidence on incorrect predictions. Conformal prediction bypasses this by providing distribution-free coverage guarantees that do not depend on calibration quality.

A quantitative evaluation framework that measures the factual consistency of a generated summary or answer relative to the source material. In legal contexts, faithfulness directly addresses hallucination by detecting contradictions and unsupported fabrications.

• Entailment-based metrics use NLI models to check if each generated claim is logically supported by the source
• Question-answering based metrics generate questions from the output and verify answers against the source
• n-gram overlap metrics like ROUGE are necessary but insufficient for detecting subtle hallucinations

Conformal prediction complements faithfulness metrics by providing a statistical wrapper around any classifier, guaranteeing that the true label falls within the prediction set at a user-specified rate.

The automated process of verifying that every factual claim in a generated text is explicitly supported by the provided source document. This serves as a critical guardrail against hallucination in legal AI systems.

• Claim decomposition breaks generated text into atomic factual assertions
• Evidence retrieval maps each claim to the most relevant source passage
• Entailment classification determines if the source passage logically supports the claim

While groundedness detection provides a binary or graded signal of support, conformal prediction adds a formal coverage guarantee. Together, they form a defense-in-depth strategy: groundedness checks content, while conformal prediction controls statistical error rates.

A classification task that determines whether a hypothesis can be logically inferred from a premise. In legal AI, NLI is the computational engine behind citation verification and hallucination detection.

• Entailment: the premise logically supports the hypothesis
• Contradiction: the premise directly refutes the hypothesis
• Neutral: the premise provides insufficient information

State-of-the-art NLI models like DeBERTa-v3 fine-tuned on the Multi-NLI corpus achieve over 91% accuracy. When paired with conformal prediction, NLI-based verifiers can produce prediction sets with formal guarantees on the correctness of entailment judgments.

A metric that quantifies the degree to which a generated statement can be directly linked to a specific segment of a source document. This ensures every legal conclusion has a verifiable provenance.

• Token-level attribution maps each generated token to its most influential source token using gradient-based methods
• Sentence-level attribution uses similarity search to find the closest source passage
• Citation-based attribution verifies that explicit references genuinely support the associated claim

Attribution scoring provides traceability—the audit trail from output back to source. Conformal prediction provides reliability—the statistical guarantee that the attribution is correct at a specified confidence level. Both are essential for high-stakes legal applications.