Variational Inference

Unlike Markov Chain Monte Carlo (MCMC) methods that rely on sampling, VI frames inference as a deterministic optimization problem. The goal is to find the member of a simpler, tractable family of distributions $Q$ that is closest to the true posterior $P$. This is achieved by minimizing the Kullback-Leibler (KL) divergence between the approximate and true distributions, making VI highly scalable to massive datasets where sampling is computationally prohibitive.

Directly minimizing the KL divergence is impossible because it requires the intractable true posterior. VI sidesteps this by maximizing the Evidence Lower Bound (ELBO). The ELBO is a lower bound on the log marginal likelihood of the data. Maximizing the ELBO simultaneously minimizes the KL divergence to the posterior and maximizes the data fit. The ELBO decomposes into:

• Expected log-likelihood: Rewards distributions that explain the observed data well.
• KL divergence to the prior: Penalizes distributions that stray too far from the prior, acting as a built-in regularizer to prevent overfitting.

The most common tractable family is the mean-field variational family, which assumes that all latent variables are mutually independent. This factorization breaks the complex dependencies in the true posterior, allowing the joint approximation to be expressed as a product of individual factors. While this assumption introduces bias and underestimates posterior variance, it enables the derivation of simple coordinate ascent update rules, making the optimization analytically tractable and computationally efficient for models like Latent Dirichlet Allocation (LDA).

Modern VI scales to deep neural networks using stochastic gradient ascent on the ELBO. Two dominant approaches exist:

• Score Function Gradient (REINFORCE): A black-box method that estimates the gradient using the log-derivative trick. It is general but suffers from high variance.
• Reparameterization Trick: Applicable when the latent variable can be expressed as a deterministic transformation of a noise variable. This allows the gradient to flow directly through the sampling process, yielding low-variance estimates and enabling Variational Autoencoders (VAEs) to be trained efficiently with standard backpropagation.

Instead of optimizing separate variational parameters for each data point, amortized inference uses a shared neural network (an inference network or encoder) to map an observation directly to its corresponding approximate posterior parameters. This 'amortizes' the cost of inference across the dataset. The inference network learns a global mapping from data to latent distributions, enabling fast inference for unseen data points without re-running the optimization loop. This is the foundational principle behind the encoder in a Variational Autoencoder (VAE).

VI introduces a systematic bias because the true posterior is restricted to lie within the chosen variational family. If the family is too simple (e.g., a diagonal Gaussian), the approximation cannot capture complex correlations, leading to an underdispersed posterior. However, this bias is traded for a massive reduction in variance compared to MCMC. VI provides a deterministic, smooth approximation that is computationally stable and fast, making it the preferred tool when a fast, closed-form approximation is more valuable than an asymptotically exact sample.

The factorization assumption that makes variational inference computationally tractable. Mean-field variational families assume all latent variables are mutually independent, breaking the joint posterior into a product of individual distributions:

• Each variable gets its own variational factor.
• Dependencies between latent variables are deliberately ignored.
• Enables coordinate ascent updates via closed-form expressions.

While computationally efficient, this assumption introduces bias by failing to capture posterior correlations, leading to overconfident uncertainty estimates in models with tightly coupled latent structure.

The initial belief about latent variable values before observing any data, encoded as a probability distribution. In variational inference, the prior acts as a regularizer through the KL divergence term in the ELBO:

• Conjugate priors: Enable closed-form variational updates.
• Non-conjugate priors: Require black-box or Monte Carlo gradient estimators.
• Hierarchical priors: Allow sharing statistical strength across groups.

The choice of prior strongly influences posterior behavior, especially in data-scarce regimes where the KL term dominates the ELBO objective.