Evidence Lower Bound (ELBO)

The ELBO is mathematically decomposed into two key terms:

• Expected Log-Likelihood: Measures how well the variational distribution $q_\phi(z|x)$ explains the observed data $x$. A higher value means better reconstruction.
• KL Divergence Regularizer: $D_{KL}(q_\phi(z|x) ,||, p(z))$. This term penalizes the variational posterior for deviating from the prior $p(z)$, enforcing a compact and regularized latent space.

Maximizing the ELBO is equivalent to maximizing the data likelihood (via the first term) while minimizing the divergence from the prior (via the second term). This is the variational inference objective.

This term, $\mathbb{E}{q\phi(z|x)}[\log p_\theta(x|z)]$, evaluates the generative model's ability to reconstruct the input data $x$ from the latent variable $z$ sampled from the variational posterior.

• Function: It acts as a reconstruction loss, similar to the loss in an autoencoder. For continuous data, it often corresponds to a mean-squared error; for discrete data, it corresponds to cross-entropy.
• Interpretation: A high value indicates the latent codes $z$ are informative enough for the decoder $p_\theta(x|z)$ to accurately reproduce the input.
• Challenge: Computing this expectation requires sampling from $q_\phi(z|x)$, which is addressed by the reparameterization trick to allow gradient-based optimization.

This term, $D_{KL}(q_\phi(z|x) ,||, p(z))$, measures the Kullback-Leibler Divergence between the approximate posterior and the latent prior.

• Function: It acts as a regularizer, preventing the variational posterior from collapsing to a point mass and encouraging the latent representations to conform to the prior structure (e.g., a standard Gaussian $\mathcal{N}(0, I)$).
• Effect: It trades off reconstruction accuracy for a more structured, disentangled, or compressed latent space. Without it, the model could overfit or learn trivial latent codes.
• Closed Form: For common choices like Gaussian $q_\phi$ and $p(z)$, this term can be computed analytically, leading to stable training.

A critical technique for enabling gradient-based optimization of the ELBO with respect to the variational parameters $\phi$.

• Problem: The gradient of the expectation $\nabla_\phi \mathbb{E}{q\phi(z|x)}[\cdot]$ is intractable to compute directly due to the dependence of the distribution on $\phi$.
• Solution: Parameterize the random variable $z$ as a deterministic function of $\phi$ and a noise variable $\epsilon$ sampled from a fixed distribution. For a Gaussian $q_\phi$, this is: $z = \mu_\phi + \sigma_\phi \odot \epsilon$, where $\epsilon \sim \mathcal{N}(0, I)$.
• Result: The gradient can now flow through this deterministic path, allowing standard backpropagation to optimize both the encoder ($\phi$) and decoder ($\theta$) parameters simultaneously.

The ELBO decomposition has a direct interpretation through the lens of information theory, framing variational inference as a rate-distortion problem.

• Distortion: The negative expected log-likelihood term. It measures the loss (distortion) incurred when reconstructing $x$ from the latent code $z$.
• Rate: The KL divergence term. It measures the number of nats (information units) required to transmit the latent code $z$ using a code optimized for the prior $p(z)$.
• Trade-off: Maximizing the ELBO is equivalent to minimizing the distortion for a given rate, or finding the most efficient latent representation that preserves essential information about the data.

The gap between the true log-evidence $\log p(x)$ and the ELBO is exactly the KL divergence between the approximate and true posterior: $\log p(x) - \text{ELBO} = D_{KL}(q_\phi(z|x) ,||, p(z|x))$.

• Implication: The ELBO becomes a tighter (better) lower bound as the variational distribution $q_\phi(z|x)$ more closely approximates the true, intractable posterior $p(z|x)$.
• Goal of Optimization: Maximizing the ELBO directly minimizes this gap, forcing $q_\phi$ toward $p(z|x)$. At the global optimum (if reachable), $q_\phi(z|x) = p(z|x)$ and the ELBO equals the true log-evidence.
• Practical Limit: The flexibility of the variational family (e.g., a diagonal Gaussian) often limits how tight the bound can become, creating an approximation gap.