Evidence Lower Bound (ELBO)

The ELBO is mathematically decomposed into two opposing terms that define its optimization landscape:

• Expected Log-Likelihood (Reconstruction Error): This term maximizes the probability of observing the data x given the latent variable z. It forces the approximate posterior q(z|x) to encode meaningful representations that can accurately reconstruct the input.
• Kullback-Leibler (KL) Divergence (Regularization): This term measures the information loss when using the approximate posterior q(z|x) instead of the true prior p(z). It acts as a complexity penalty, preventing the posterior from collapsing to point estimates and keeping it close to the prior distribution. The balance between these two terms prevents overfitting and controls the information bottleneck in models like Variational Autoencoders (VAEs).

The ELBO is derived from the intractable log marginal likelihood log p(x) using Jensen's Inequality:

1. Start with the marginal: log p(x) = log ∫ p(x, z) dz
2. Introduce the variational distribution: Multiply and divide by q(z|x).
3. Apply Jensen's Inequality: Since the logarithm is a concave function, log(E[X]) ≥ E[log(X)]. This yields: log p(x) ≥ E_q[log p(x, z) - log q(z|x)] The right-hand side is the ELBO. The gap between the true marginal likelihood and the ELBO is exactly the KL divergence KL(q(z|x) || p(z|x)). Maximizing the ELBO implicitly minimizes this divergence, tightening the bound.

The ELBO is called a 'lower bound' because it is always less than or equal to the true log evidence log p(x). The tightness of this bound is critical:

• Exact Gap: log p(x) = ELBO + KL(q(z|x) || p(z|x))
• Perfect Fit: If the variational family q(z|x) is flexible enough to capture the true posterior p(z|x) exactly, the KL divergence becomes zero, and the ELBO equals the log evidence.
• Approximation Error: In practice, the choice of the variational family (e.g., a factorized Gaussian) restricts the achievable tightness. A restrictive family introduces a non-zero gap, resulting in a biased posterior approximation even at the optimum.

In a VAE, the ELBO is the objective function used to jointly train an inference network (encoder) and a generative network (decoder):

• Encoder q_φ(z|x): Parameterizes the approximate posterior, predicting a mean and variance for the latent Gaussian.
• Decoder p_θ(x|z): Parameterizes the likelihood, reconstructing the input from a latent sample.
• The Reparameterization Trick: To backpropagate through the stochastic sampling step z ~ q_φ(z|x), the random variable is expressed as a deterministic transformation of a noise variable ε (e.g., z = μ + σ * ε). This allows gradients to flow from the reconstruction loss to the encoder parameters φ.

The β-VAE introduces a hyperparameter β to the ELBO to control the trade-off between reconstruction fidelity and latent channel capacity: ELBO = E_q[log p(x|z)] - β * KL(q(z|x) || p(z))

• β = 1: Standard VAE formulation.
• β > 1: Applies a stronger constraint on the latent information bottleneck. This forces the model to learn more efficient, factorized latent representations where individual dimensions correspond to independent, interpretable generative factors.
• Trade-off: High β values encourage disentanglement but often degrade reconstruction sharpness because the latent code lacks the capacity to store fine details.

The ELBO is typically optimized using stochastic gradient descent on minibatches of data. A key practical challenge is the high variance of the gradient estimator:

• Score Function Estimator (REINFORCE): A generic but high-variance method that does not require a differentiable density.
• Reparameterization Trick: The standard for continuous latent variables, providing a lower-variance gradient estimator by moving the stochasticity to an input noise variable.
• Control Variates: Variance reduction techniques, such as the STL estimator or VIMCO, are often used to stabilize training, especially for discrete latent variables where the reparameterization trick is not directly applicable.

The ELBO is the loss function that defines the VAE framework. The reconstruction loss term maximizes the expected log-likelihood of the data given the latent variables, forcing the decoder to generate realistic samples. The KL divergence term acts as a regularizer, pulling the learned approximate posterior towards a chosen prior (typically a standard Gaussian). This dual optimization creates a smooth, continuous latent space that enables high-quality generative sampling and interpolation.

In BNNs, the ELBO provides a tractable objective for learning distributions over network weights instead of point estimates. The process, known as Bayes by Backprop, minimizes the KL divergence between a variational posterior over weights and a prior. This directly quantifies epistemic uncertainty in predictions. Maximizing the ELBO here means finding a distribution over weights that explains the training data well while remaining close to our prior beliefs, preventing overconfidence on out-of-distribution inputs.

The ELBO enables learning an inference network that maps observations directly to approximate posterior parameters in a single forward pass. Instead of running an iterative optimization for every new data point, the inference network is trained across the entire dataset to maximize the expected ELBO. This amortization is the core of the VAE's encoder and is critical for scaling variational inference to large datasets where per-sample optimization would be computationally prohibitive.

The ELBO serves as a principled criterion for model comparison in Bayesian frameworks. Since it lower-bounds the log model evidence (marginal likelihood), a higher ELBO indicates a model that better explains the observed data while automatically penalizing unnecessary complexity via the KL term. This inherent Occam's razor effect is used for automatic relevance determination, where optimizing the ELBO drives irrelevant weight parameters towards zero, effectively pruning the network without a separate validation set.

By modifying the ELBO with a controlled KL divergence weight, such as in the β-VAE framework, the objective can be tuned to prioritize the KL term. A β > 1 applies stronger pressure to match the factorial Gaussian prior, encouraging the latent dimensions to encode statistically independent generative factors. Maximizing this modified ELBO yields a latent space where individual dimensions correspond to interpretable, disentangled features like object position, scale, or lighting in images.

The ELBO is optimized in practice using the SGVB estimator, which provides a differentiable, low-variance Monte Carlo gradient estimate. By applying the reparameterization trick, the random sampling node is moved outside the computational graph, allowing standard backpropagation. This transforms the abstract problem of inference into a concrete optimization task solvable with stochastic gradient descent, making variational inference scalable to the massive models and datasets that define modern deep learning.

An optimization-based technique that approximates complex posterior distributions by finding the closest distribution within a simpler, tractable family. The ELBO serves as the objective function to be maximized, minimizing the KL divergence between the approximate and true posterior without requiring the intractable marginal likelihood. This transforms Bayesian inference from a sampling problem into an optimization problem.

A neural network where weights are treated as probability distributions rather than point estimates, enabling principled uncertainty quantification. Training a BNN typically involves maximizing the ELBO to learn the parameters of the weight distributions. The resulting model can distinguish between epistemic uncertainty (model ignorance) and aleatoric uncertainty (data noise), critical for high-stakes deployment.

The updated belief about a model parameter's value after combining the prior distribution with observed data via Bayes' theorem. Computing the exact posterior is often intractable for deep models. The ELBO provides a tractable lower bound that, when maximized, yields an approximate posterior that balances fidelity to the data against proximity to the prior, acting as a regularized fit.

The initial belief about a model parameter's value before observing any data, encoded as a probability distribution. In the ELBO decomposition, the KL divergence term acts as a regularizer that penalizes the approximate posterior for deviating too far from the prior. Common choices include standard Gaussian priors for weight distributions in variational BNNs, encouraging simplicity and preventing overfitting.

Model uncertainty arising from a lack of knowledge or data, which can theoretically be reduced with more training samples or a better model architecture. Maximizing the ELBO in a Bayesian framework captures this uncertainty in the spread of the weight posterior. High epistemic uncertainty signals that the model has not seen enough data in a region of input space to make a confident prediction.

Inherent and irreducible noise in the data generation process, such as measurement error or inherent stochasticity. Unlike epistemic uncertainty, this cannot be reduced by collecting more data. The ELBO framework can be extended to model aleatoric uncertainty by outputting the parameters of a noise distribution (e.g., the variance of a Gaussian) alongside the prediction, learning to quantify the intrinsic data noise.