Conformal Prediction

The core theorem of conformal prediction provides a finite-sample, distribution-free guarantee. If the calibration and test data are exchangeable, the probability that the true label is included in the prediction set is at least the user-specified confidence level (1 - α).

• Mathematical guarantee: P(Y_test ∈ C(X_test)) ≥ 1 - α
• Holds regardless of the underlying model or data distribution
• Valid for any sample size, not just asymptotically
• Assumes only exchangeability of calibration and test points

The mechanism that drives conformal prediction is the nonconformity score, a heuristic measure of how unusual a potential label is for a given input. The framework is agnostic to the specific scoring function.

• Regression: Often uses absolute residual |y - ŷ| as the score
• Classification: Typically uses 1 - softmax probability of the true class
• The score quantifies the model's 'surprise' at a given (x, y) pair
• Better scores yield tighter, more informative prediction sets

The most practical variant that avoids retraining the model. The dataset is split into a proper training set and a calibration set, making it computationally efficient for deep learning.

• Step 1: Train the model on the proper training set
• Step 2: Compute nonconformity scores on the held-out calibration set
• Step 3: Determine the empirical quantile threshold from calibration scores
• Step 4: For new test points, include all labels with scores below the threshold
• Avoids the computational cost of full (transductive) conformal prediction

Unlike naive constant-width confidence intervals, conformal prediction produces adaptive sets that shrink when the model is confident and expand when uncertain. This provides actionable uncertainty information.

• Easy examples: Small prediction sets, often a single class or narrow interval
• Ambiguous examples: Larger sets reflecting genuine uncertainty
• Out-of-distribution inputs: Very large or empty sets signal distribution shift
• Enables risk-based decision making by flagging unreliable predictions

Standard conformal prediction guarantees marginal coverage (averaged over all test points), not conditional coverage per subgroup. Achieving conditional validity is an active research area.

• Marginal: Coverage holds on average across the population
• Conditional: Coverage holds within each stratum (e.g., per class, per demographic)
• Mondrian conformal prediction extends the framework to achieve class-conditional coverage
• Important for fairness: ensures guarantees hold across protected subgroups

An extension of conformal prediction that provides guarantees on arbitrary loss functions, not just coverage. This generalizes the framework beyond set-valued predictions to tasks like bounding false negative rates.

• Controls the expected value of any monotone loss function
• Applications: FDR control in multiple testing, bounding segmentation errors
• Uses a similar calibration-quantile mechanism with a more general formulation
• Enables rigorous guarantees for structured prediction tasks

Instead of a single point prediction, conformal prediction outputs a prediction set containing multiple plausible outputs. The framework guarantees that the true output is included in this set at a user-specified confidence level (e.g., 95%).

• Calibration Step: Uses a held-out calibration dataset to learn a nonconformity score threshold.
• Inference Step: For a new input, includes all outputs with nonconformity scores below the threshold.
• Key Guarantee: Marginal coverage probability is mathematically guaranteed, assuming only exchangeability of data points.

The nonconformity score measures how unusual a prediction is relative to the calibration data. For LLM hallucination detection, several scoring functions are effective:

• Semantic Entropy: Clusters semantically equivalent generations before calculating entropy, distinguishing genuine uncertainty from lexical variation.
• SelfCheckGPT Score: Measures the inconsistency across multiple sampled responses; hallucinated facts exhibit high stochastic variability.
• NLI Entailment Score: Uses a Natural Language Inference model to score how well a generated claim is entailed by a retrieved grounding document.
• Token-Level Probability: Uses the model's own softmax probabilities as a baseline nonconformity measure.

Conformal prediction enables risk-controlling prediction sets that directly bound the hallucination rate. A risk manager can specify a desired error tolerance (e.g., 5% hallucination rate) and the framework automatically adjusts the prediction set size to meet this guarantee.

• Selective Prediction: The model abstains from answering when the prediction set is empty or exceeds a size threshold, flagging high-uncertainty inputs for human review.
• Adaptive Thresholding: The nonconformity threshold adapts to input difficulty, producing larger sets for ambiguous queries and tighter sets for well-grounded ones.
• False Discovery Rate Control: Extensions like conformal risk control bound the expected proportion of hallucinated claims in a batch of generations.

Conformal prediction enhances Retrieval-Augmented Generation by providing statistical guarantees on grounding quality:

• Retrieval Set Calibration: Produces a conformal set of retrieved documents guaranteed to contain the relevant passage at a specified confidence level.
• Claim-Level Verification: Each atomic fact in a generated response receives a conformal p-value indicating the likelihood of hallucination relative to the retrieved context.
• Dynamic Fallback Triggering: When the conformal prediction set for a generated claim fails to meet the coverage guarantee, the system automatically triggers a re-retrieval or invokes a fact-checking module.
• Citation Quality Guarantee: Ensures that cited sources actually support the generated claims with a statistically bounded error rate.

In conversational AI, hallucination risk compounds across multiple turns. Conformal risk control extends the framework to sequential decision-making:

• Turn-Level Guarantees: Controls the expected hallucination rate per conversational turn, preventing error propagation.
• Session-Level Bounds: Provides cumulative error guarantees over an entire dialogue session using martingale-based methods.
• Adaptive Abstention: The system learns to abstain or request clarification when conformal bounds indicate insufficient grounding, maintaining the coverage guarantee.
• Online Calibration: Continuously updates nonconformity thresholds as new user feedback or corrections are received during the conversation.

Conformal prediction provides rigorous evaluation metrics beyond simple accuracy for hallucination detection systems:

• Conformal AUROC: Measures the area under the ROC curve for conformal p-values, assessing how well the method separates hallucinated from truthful claims.
• Set Size Efficiency: Evaluates the average size of prediction sets at a fixed confidence level; smaller sets indicate more precise uncertainty quantification.
• Conditional Coverage Diagnostics: Tests whether coverage guarantees hold across different data subgroups (e.g., short vs. long inputs, common vs. rare entities).
• Distribution Shift Robustness: Quantifies how coverage guarantees degrade under covariate shift, critical for production deployments where input distributions evolve.

The field of machine learning focused on estimating the confidence bounds of a model's predictions to identify when the model is likely to be wrong, enabling risk-based decision making. Conformal prediction is a specific distribution-free UQ technique.

• Distinguishes between epistemic uncertainty (reducible, from lack of knowledge) and aleatoric uncertainty (irreducible, from data noise)
• Foundational for high-stakes applications like medical diagnosis and autonomous driving
• Complements conformal prediction by providing the theoretical framework for why prediction sets are necessary

A scalar summary statistic that measures the miscalibration of a model by computing the weighted average of the difference between confidence and accuracy across discrete probability bins.

• A perfectly calibrated model with 90% confidence should be correct 90% of the time
• ECE bins predictions by confidence level and compares each bin's average confidence to its actual accuracy
• While conformal prediction guarantees marginal coverage, ECE evaluates the quality of the underlying probability estimates themselves

A measure of uncertainty in language model outputs that clusters semantically equivalent generations before calculating entropy, distinguishing between high uncertainty and simple lexical variation.

• Unlike naive token-level entropy, semantic entropy groups paraphrases like "Paris is the capital of France" and "France's capital is Paris" as equivalent
• High semantic entropy strongly correlates with hallucination risk
• Used alongside conformal prediction to identify when a model is guessing rather than retrieving facts

A zero-resource hallucination detection method that samples multiple responses from a black-box LLM and checks for factual inconsistency, leveraging the idea that hallucinated facts are stochastically unstable.

• Requires no external knowledge base or training data
• Operates on the principle that consistent facts appear across multiple samples while hallucinations vary randomly
• Complements conformal prediction by providing a lightweight, post-hoc check on generation reliability without requiring model internals

A technique for quantifying predictive uncertainty by training multiple independent models with different random initializations on the same data and measuring the variance of their predictions.

• High variance across ensemble members indicates high epistemic uncertainty
• Provides a Bayesian approximation without requiring explicit Bayesian inference
• Unlike conformal prediction's distribution-free guarantee, deep ensembles rely on the assumption that model diversity captures meaningful uncertainty

A programmable framework or set of rules that sits between a user and an LLM to intercept, validate, and correct outputs in real-time, enforcing structural and factual constraints to prevent hallucinations.

• NeMo Guardrails by NVIDIA enables defining topical, safety, and factual verification rails
• Can integrate conformal prediction sets to reject outputs that fall outside acceptable confidence bounds
• Operates as a runtime enforcement layer rather than a statistical guarantee mechanism