Conformalized LLM

Instead of a single point prediction, a conformalized LLM outputs a prediction set of possible answers. The core guarantee is that the true, correct answer is contained within this set with a user-specified probability (e.g., 95%).

• Mechanism: A nonconformity score measures how 'unusual' a candidate answer is based on a held-out calibration set.
• Result: The set dynamically sizes itself—returning a single answer for high-confidence inputs and a larger set for ambiguous ones.
• Example: For a medical question, the set might be {ibuprofen, acetaminophen} rather than a hallucinated single drug.

Conformal risk control extends the framework to directly bound the false discovery rate in generated text. This provides a statistical lever to control hallucinations in open-ended generation tasks.

• Loss Function: Define a monotone loss, such as the number of incorrect factual statements in a summary.
• Calibration: A parameter λ is tuned on a calibration set to ensure the expected loss stays below a target threshold α.
• Practical Impact: An enterprise RAG system can guarantee that generated summaries contain, on average, fewer than one hallucinated entity per document.

Conformal prediction enables a model to abstain from answering when it is uncertain, rather than guessing. The model outputs a prediction set that may be empty or contain a special 'I don't know' token.

• Nonconformity Threshold: If no candidate answer's score falls below the calibrated threshold, the set is empty.
• Coverage Guarantee: The marginal guarantee still holds—the true label is in the set when the set is non-empty, with the specified probability.
• Enterprise Value: Prevents automated customer-facing agents from providing incorrect information during ambiguous queries.

For retrieval-augmented generation, conformal prediction can guarantee that the set of retrieved documents contains all relevant information. This is a set-valued retrieval approach.

• Nonconformity Measure: Based on the semantic similarity between the query and a document, calibrated against a set of known relevant passages.
• Guarantee: The retrieval set is guaranteed to contain the true supporting document with probability 1 - α.
• Application: Legal document review where missing a single relevant precedent is a critical failure.

Standard conformal prediction provides only marginal coverage, which can fail for specific subgroups. Mondrian conformal prediction applies calibration independently within pre-defined categories.

• Stratification: The calibration set is partitioned by topic, language, or user demographic.
• Guarantee: Coverage holds conditionally for each group, e.g., 95% coverage for both English and Japanese queries.
• Fairness: Prevents a model from being overconfident on a majority language while hedging excessively on a minority one.

Adaptive conformal inference (ACI) adjusts the prediction set size in real-time to maintain coverage as the distribution of user queries shifts over time.

• Mechanism: The quantile threshold is increased or decreased based on recent miscoverage events.
• No Retraining: The underlying LLM remains frozen; only the calibration threshold is updated online.
• Use Case: A financial sentiment analysis LLM that must remain valid during a sudden market crash when language patterns change abruptly.