Deep Ensemble Uncertainty

The core mechanism separates epistemic uncertainty (model ignorance) from aleatoric uncertainty (data noise).

• High Variance: When models in the ensemble produce wildly different predictions for the same input, the ensemble has high epistemic uncertainty—it doesn't know the correct answer.
• Low Variance: When models agree, epistemic uncertainty is low, even if the average prediction is still probabilistic due to inherent data noise.
• This decomposition allows a system to distinguish between 'I don't know because I haven't seen this before' and 'The data itself is ambiguous.'

The power of a deep ensemble comes from training identical architectures with different random seeds. This exploits the non-convex loss landscape of neural networks.

• Each model converges to a different local minimum, a functionally distinct solution.
• These distinct solutions form a distribution over functions, not just parameters.
• The disagreement between these modes in the under-sampled regions of the input space provides a robust signal for out-of-distribution (OOD) detection.

Deep Ensembles are a parallelizable but computationally expensive alternative to single-model Bayesian approximations like Monte Carlo (MC) Dropout.

• Deep Ensembles: Train M models from scratch. Cost is M times the training budget, but inference can be batched efficiently.
• MC Dropout: Performs T stochastic forward passes through a single model. Training cost is low, but it often produces less calibrated uncertainty estimates.
• Empirical studies show Deep Ensembles consistently outperform MC Dropout on predictive entropy and mutual information metrics for OOD detection.

Ensemble disagreement is a powerful signal for detecting inputs that lie outside the training distribution, including adversarial examples.

• Adversarial Detection: An input crafted to fool one model is unlikely to simultaneously fool all M models in the same way. High variance in the ensemble's output layer signals an attack.
• Novelty Detection: For a truly novel input class, each model will extrapolate differently, resulting in a high-variance, low-confidence prediction.
• This makes deep ensembles a practical defense layer in production ML security.

Deep Ensembles are trained with a proper scoring rule, typically the negative log-likelihood (NLL), which encourages calibrated confidence estimates.

• Calibration: The ensemble's average confidence should match its accuracy. A 90% confidence prediction should be correct 90% of the time.
• Ensemble Averaging: Averaging the softmax probabilities from M models naturally smooths overconfident, spiky predictions from individual members.
• This results in a lower Expected Calibration Error (ECE) compared to single-model baselines.

To reduce the inference cost of running M models in production, the ensemble's aggregated knowledge can be distilled into a single student model.

• Training: A single student network is trained to mimic the average predictive distribution of the full ensemble, not just the hard labels.
• Trade-off: This captures the mean prediction well but loses the variance information, making it impossible to compute epistemic uncertainty at inference time.
• This technique is used when the goal is a single, highly accurate, and calibrated model, rather than an uncertainty-aware system.

Deep ensembles primarily capture epistemic uncertainty—the reducible ignorance from limited data or model capacity. This is distinct from aleatoric uncertainty, which is the irreducible noise inherent in the data itself (e.g., ambiguous labels, sensor noise).

• Epistemic: High in sparse regions of training data; decreases with more data.
• Aleatoric: Constant regardless of dataset size; requires output distribution modeling.
• Ensemble variance isolates epistemic uncertainty, signaling when the model should defer to a human.

A scalar metric that quantifies how well a model's predicted confidence aligns with its actual accuracy. ECE bins predictions by confidence level and computes the weighted average of the gap between confidence and accuracy.

• A perfectly calibrated model with 90% confidence is correct 90% of the time.
• Modern neural networks are notoriously overconfident, exhibiting high ECE.
• Deep ensembles naturally improve calibration, often achieving lower ECE than single models or Monte Carlo Dropout.

A method for detecting hallucinations by clustering semantically equivalent generations before calculating entropy. Unlike naive token-level entropy, this distinguishes genuine uncertainty from simple lexical variation.

• Generates multiple samples from the same prompt.
• Groups outputs with identical meaning using Natural Language Inference.
• High semantic entropy indicates the model is vacillating between contradictory facts—a strong hallucination signal.
• Complements ensemble variance by operating on a single model's output distribution.

A distribution-free statistical framework that wraps any model to produce prediction sets with rigorous finite-sample guarantees. Unlike Bayesian methods, it makes no assumptions about the underlying data distribution.

• Given a user-specified confidence level (e.g., 95%), it outputs a set containing the true label with at least that probability.
• The size of the prediction set serves as an uncertainty measure: larger sets indicate higher uncertainty.
• Can be applied post-hoc to deep ensemble outputs for guaranteed coverage.

A zero-resource hallucination detection technique that leverages the stochastic nature of LLMs. The core insight: hallucinated facts are inconsistent across independent samples.

• Samples multiple responses to the same prompt from a black-box model.
• Compares each new response against the others using metrics like BERTScore or an NLI entailment model.
• Low agreement across samples indicates a likely hallucination.
• Functions without external knowledge bases, making it applicable to any black-box API.

A prompting strategy where an LLM systematically fact-checks its own output before presenting it to the user. The model executes a multi-stage pipeline:

1. Draft: Generate an initial response.
2. Verify: Produce a set of independent verification questions based on the draft.
3. Answer: Answer those questions without referencing the draft to avoid bias.
4. Correct: Revise the original response to align with verified answers.

This approach reduces hallucinations in long-form generation without external tools.