Bayesian Neural Networks

The core feature of a BNN is its ability to produce predictive uncertainty estimates alongside its predictions. Instead of a single output, a BNN outputs a predictive distribution. This is decomposed into two key types of uncertainty:

• Aleatoric Uncertainty: Irreducible noise inherent in the data (e.g., sensor noise). The model learns to estimate this from the data.
• Epistemic Uncertainty: Model uncertainty due to a lack of knowledge, which can be reduced with more data. This is captured by the distribution over weights.

This allows for risk-aware decision-making, flagging low-confidence predictions where the model is uncertain, which is critical for safety-critical applications like medical diagnosis or autonomous driving.

BNNs replace the deterministic, point-estimate weights of standard neural networks with probability distributions. Each weight is not a single number but a distribution (e.g., Gaussian) characterized by a mean and variance.

• Prior Distribution: Before seeing data, weights are initialized with a prior distribution (e.g., a standard normal), representing initial beliefs.
• Posterior Distribution: After training on data, Bayes' theorem is used to infer the posterior distribution over weights, which represents updated beliefs.

This probabilistic formulation is the mathematical foundation that enables uncertainty estimation and inherent regularization, as the model is penalized for being overly certain without evidence.

BNNs perform Bayesian inference to compute the posterior distribution. Since the true posterior is intractable for deep networks, approximate inference methods are used:

• Variational Inference (VI): Approximates the true posterior with a simpler, tractable distribution (the variational posterior) by minimizing the Kullback-Leibler (KL) divergence.
• Markov Chain Monte Carlo (MCMC): Uses sampling methods (e.g., Hamiltonian Monte Carlo) to draw samples from the true posterior, often more accurate but computationally expensive.

Predictions are made through marginalization, integrating over all possible weights according to the posterior: p(y|x, D) = ∫ p(y|x, w) p(w|D) dw. This integration, often approximated via sampling, is what produces the predictive distribution.

The Bayesian framework provides a principled form of regularization that helps prevent overfitting. The KL divergence term in the variational inference objective (the Evidence Lower Bound or ELBO) acts as a regularizer:

ELBO = Expected Log Likelihood - KL( Q(w) || P(w) )

This term penalizes the variational posterior Q(w) for deviating too far from the prior P(w). It encourages the model to find simpler explanations for the data unless the evidence strongly supports complexity, akin to weight decay in deterministic networks but derived from probability theory. This makes BNNs robust even with limited data.

A groundbreaking and widely adopted practical method for approximating BNNs is Monte Carlo Dropout. It was shown that training a standard neural network with dropout and applying dropout at test time is equivalent to performing approximate variational inference in a specific Bayesian model.

• Procedure: Enable dropout during inference and perform multiple forward passes (T passes, e.g., T=30).
• Result: The variance across the T different stochastic outputs approximates the model's (epistemic) predictive uncertainty.

This method provides a computationally cheap way to gain uncertainty estimates without changing the standard training pipeline significantly, making Bayesian principles accessible for production models.

Deep Ensembles are a powerful, non-Bayesian alternative for uncertainty estimation. Key contrasts with BNNs:

• Mechanism: Trains multiple deterministic models from different random initializations and aggregates their predictions.
• Uncertainty Source: Captures uncertainty from model initialization and optimization trajectory, which can approximate a mixture of Gaussians in function space.
• Performance: Often achieves stronger predictive accuracy and uncertainty calibration than many approximate BNN methods.
• Cost: Requires training and storing M independent models (e.g., M=5), leading to M times the computational cost for training and inference, whereas a single BNN model can generate multiple predictions via sampling.

Both are considered state-of-the-art for uncertainty, with ensembles often leading in benchmarks but BNNs offering a more unified probabilistic framework.

Deep ensembles are a practical and highly effective method for uncertainty quantification that involves training multiple deterministic neural networks with different random initializations on the same dataset. Predictions are aggregated (e.g., via mean and variance) across the ensemble.

• Mechanism: Relies on model uncertainty (epistemic) captured by the disagreement between ensemble members.
• Contrast with BNNs: While both estimate uncertainty, deep ensembles use point-estimate weights and do not explicitly model a posterior distribution over parameters. They are often simpler to implement and can outperform approximate Bayesian methods.
• Use Case: A common baseline for uncertainty-aware machine learning in computer vision and regression tasks.

Monte Carlo Dropout is a technique that approximates Bayesian inference in neural networks by using dropout—typically a regularization method—during both training and inference. Multiple stochastic forward passes are performed to generate a distribution of predictions.

• Mechanism: Framed as performing approximate variational inference, where the dropout mask distribution represents a variational distribution over the model weights.
• Relation to BNNs: Provides a computationally cheap way to turn a standard neural network into a practical, approximate BNN without changing the training objective. It is a specific instance of a Bayesian neural network with a Bernoulli variational posterior.
• Application: Enables uncertainty estimates from pre-trained models with minimal modification, useful for active learning and out-of-distribution detection.

Variational Inference is the core optimization framework used to make Bayesian neural networks tractable. It approximates the true, intractable posterior distribution over weights with a simpler, parameterized distribution (e.g., a Gaussian).

• Mechanism: Minimizes the Kullback-Leibler (KL) divergence between the approximate variational distribution and the true posterior. This is equivalent to maximizing the Evidence Lower Bound (ELBO).
• Role in BNNs: The standard training methodology for modern BNNs. The choice of variational family (mean-field, structured, etc.) is a major design decision impacting the quality of uncertainty estimates.
• Challenge: Balancing the complexity of the approximation with computational cost; a poor variational family can lead to underestimated uncertainty.

The Laplace Approximation is a classic method for approximating a posterior distribution by fitting a Gaussian distribution centered at the maximum a posteriori (MAP) estimate, with a covariance matrix given by the inverse Hessian of the log-posterior.

• Mechanism: After training a standard neural network to find the MAP solution, the Hessian matrix (or its inverse) is computed or approximated to define the Gaussian's covariance, capturing local curvature.
• Relation to BNNs: Provides a post-hoc way to add uncertainty estimates to a pre-trained deterministic model. It is computationally challenging for large models due to the Hessian's size but benefits from modern approximations like the Generalized Gauss-Newton matrix.
• Use Case: Efficient uncertainty quantification for medium-sized models where retraining with VI is undesirable.

In Bayesian deep learning, uncertainty is decomposed into two fundamental types:

• Epistemic Uncertainty (Model Uncertainty): Arises from a lack of knowledge about the model's parameters. It is reducible with more data. Bayesian neural networks are explicitly designed to capture this.
• Aleatoric Uncertainty (Data Uncertainty): Inherent noise or randomness in the observation process (e.g., sensor noise). It is irreducible with more data but can be heteroscedastic (varying with input).

BNNs can model both by placing a distribution over weights (epistemic) and by having a probabilistic output layer (aleatoric). Understanding this distinction is critical for applications like safe reinforcement learning or medical diagnosis, where knowing why the model is uncertain guides decision-making.

Hamiltonian Monte Carlo is a Markov Chain Monte Carlo (MCMC) method that uses Hamiltonian dynamics to propose new states, allowing for efficient exploration of complex, high-dimensional posterior distributions.

• Mechanism: Uses gradient information to traverse the parameter space, reducing random walk behavior. The No-U-Turn Sampler (NUTS) is a popular, adaptive variant.
• Contrast with VI: HMC is asymptotically exact, producing samples from the true posterior, whereas VI provides an approximate distribution. However, HMC is computationally prohibitive for very large neural networks.
• Role for BNNs: Considered the "gold standard" for inference in smaller Bayesian models; used as a benchmark to evaluate the accuracy of faster variational methods in BNNs.