Deep Ensembles

The core mechanism driving ensemble performance is the non-convex loss landscape of neural networks. Each network in the ensemble is initialized with a different random seed, causing the optimizer to converge to a distinct local minimum. This functional diversity means that while all models perform well on in-distribution data, they disagree significantly on out-of-distribution inputs, directly capturing epistemic uncertainty. The variance in their predictions serves as a proxy for the model's ignorance.

Deep Ensembles naturally decompose predictive uncertainty into its constituent parts:

• Epistemic Uncertainty: Captured by the variance or disagreement among the ensemble members. High disagreement signals a lack of knowledge that could be reduced with more data.
• Aleatoric Uncertainty: Each network often outputs the parameters of a probability distribution (e.g., a Gaussian mean and variance) rather than a point estimate. The mean of these predicted variances represents the inherent, irreducible noise in the data. This decomposition is critical for distinguishing between a novel, risky input and a fundamentally noisy but known one.

Unlike Bayesian Neural Networks (BNNs) which require complex and often slow approximate inference, Deep Ensembles are embarrassingly parallel. Training is a standard, non-interacting process:

• No Communication Overhead: Each model is trained independently on the full dataset.
• Hardware Utilization: Training can be distributed across multiple GPUs or nodes with linear scaling.
• Inference: While inference cost scales linearly with the number of ensemble members, this is a predictable and manageable overhead compared to the convergence issues of variational inference methods.

A key advantage of Deep Ensembles is their superior performance under covariate shift and out-of-distribution (OOD) detection. Because each member learns a slightly different decision boundary, an anomalous input is likely to fall in a region where the models disagree. This ensemble disagreement provides a high-quality, scalar uncertainty score that reliably separates in-distribution from OOD data, often outperforming single-model uncertainty methods like Monte Carlo Dropout in benchmark tasks.

Deep Ensembles produce well-calibrated predictive distributions when trained with a proper scoring rule, such as the negative log-likelihood. The averaging of multiple probability distributions naturally smooths out overconfident predictions from any single model. This results in a lower Expected Calibration Error (ECE) and a better Brier Score compared to single deterministic networks. The ensemble's aggregated softmax output more accurately reflects the empirical frequency of correctness.

To further smooth the predictive surface and increase robustness, Deep Ensembles are often combined with adversarial training. By injecting small, worst-case perturbations during training, each network learns a locally constant loss landscape. This prevents the ensemble members from developing sharp, brittle decision boundaries that could lead to high-confidence errors. The combination of ensembling and adversarial training represents a state-of-the-art defense against both aleatoric noise and adversarial attacks.

The component of predictive uncertainty arising from model ignorance or insufficient data. Deep Ensembles primarily capture this by sampling diverse functions from the loss landscape. Key characteristics:

• Reducible with more training data
• High in regions far from training samples
• Quantified by ensemble disagreement or variance across members

The irreducible data noise inherent in the observation process. In Deep Ensembles, this is often modeled by having each member output a mean and variance (heteroscedastic loss). Key characteristics:

• Cannot be reduced with more data
• High in noisy or ambiguous inputs
• Captured by the predicted variance of individual ensemble members

A direct measure of epistemic uncertainty computed from the spread of predictions. Common metrics include:

• Variance of the predictive mean for regression
• Mutual information between model parameters and predictions for classification
• Jensen-Shannon divergence across categorical outputs High disagreement signals that the model lacks knowledge about the input region.

The process of aligning a model's predicted probability with the empirical frequency of correctness. While Deep Ensembles improve calibration over single models, they often benefit from post-hoc scaling. Key techniques:

• Temperature Scaling: A single scalar parameter T > 1 softens the softmax
• Platt Scaling: Fits a logistic regression on logits
• Ensembles naturally produce better calibrated probabilities than point estimates

The task of identifying inputs that are semantically different from the training distribution. Deep Ensembles excel here because OOD inputs cause high epistemic uncertainty (disagreement). Detection signals:

• High variance in predictive mean
• Low mutual information across ensemble members
• High entropy of the average prediction This prevents overconfident mistakes on anomalous inputs.