Ensemble Disagreement

The most direct measure of ensemble disagreement is the statistical variance across the output distributions of individual ensemble members. For regression tasks, this is computed as the variance of the predicted means. For classification, it is often measured as the variance of the predicted probability vectors. High variance indicates that different models, trained from different initializations, have learned conflicting functions for the same input region, signaling high epistemic uncertainty. This metric is zero when all models agree perfectly, regardless of whether they are correct.

A more formal information-theoretic measure decomposes total predictive uncertainty into aleatoric and epistemic components. The total entropy of the ensemble's average prediction, H[E[p(y|x)]], captures overall uncertainty. The epistemic component is measured by the mutual information between the model parameters and the prediction: I[y, θ|x] = H[E[p(y|x)]] - E[H[p(y|x)]]. This quantifies the reduction in uncertainty about the prediction if we knew the true model parameters. High mutual information indicates strong disagreement driven by model ignorance.

For classification ensembles, the Jensen-Shannon Divergence (JSD) provides a bounded, symmetric measure of disagreement between the categorical output distributions of ensemble members. Unlike variance, JSD captures multi-modal disagreement where models cluster into distinct prediction camps. A high JSD indicates that ensemble members are placing probability mass on different classes, revealing ambiguous or out-of-distribution inputs. This metric is particularly useful for detecting inputs that lie near complex decision boundaries.

Ensemble disagreement serves as a powerful out-of-distribution (OOD) detection mechanism. The core principle is that models trained on the same in-distribution data will agree on familiar inputs but diverge arbitrarily on inputs far from the training manifold. By setting a threshold on a disagreement metric like variance or mutual information, the system can flag anomalous inputs for human review or rejection. This method requires no explicit OOD training data and generalizes across architecture types.

Deep Ensembles explore distinct modes in the loss landscape by training multiple models with different random seeds, producing functionally diverse hypotheses. In contrast, Monte Carlo Dropout samples from a single model by applying dropout at inference time, approximating a variational distribution. Ensemble disagreement from deep ensembles typically provides more robust and well-calibrated uncertainty estimates because the members explore more diverse solutions, whereas MC Dropout can underestimate epistemic uncertainty in regions far from training data.

A computational bottleneck of traditional ensembles is the linear scaling of memory and FLOPs with the number of members. BatchEnsemble addresses this by sharing a single set of base weights across all ensemble members while maintaining member-specific rank-1 matrices that modulate the shared weights. This allows the ensemble to be executed as a single vectorized forward pass. The disagreement between these efficient rank-1 members still captures meaningful epistemic uncertainty while reducing the parameter overhead by orders of magnitude.