Posterior Distribution

The posterior is the result of applying Bayes' theorem, combining prior beliefs with observed evidence. Formally: p(z|x) = p(x|z)p(z) / p(x). The numerator is the product of the likelihood (how probable the data is given the latent state) and the prior (initial assumption). The denominator, the evidence or marginal likelihood, is often intractable, which is why VAEs use variational inference to approximate the posterior.

For complex models like deep neural networks, the true posterior is analytically intractable because computing the evidence p(x) requires integrating over all possible latent states. VAEs address this via variational inference, which posits a simpler, parameterized family of distributions q_φ(z|x) (the probabilistic encoder) to approximate the true posterior. The quality of this approximation is measured by the Kullback-Leibler (KL) divergence between q and p.

Instead of optimizing a separate set of variational parameters for each data point, VAEs use amortized inference. A neural network (the encoder) is trained to output the parameters (e.g., mean and variance) of the approximate posterior q_φ(z|x) for any input x. This allows for efficient, single-pass inference after training, a key scalability advantage over traditional variational methods that require per-datum optimization.

• Input: Data point x (e.g., an image).
• Output: Parameters μ_φ(x), σ_φ(x) defining a Gaussian distribution over z.

The KL divergence term in the VAE's ELBO objective regularizes the approximate posterior q_φ(z|x) towards a simple prior distribution p(z), typically a standard normal distribution N(0, I). This has several effects:

• Prevents Overfitting: Stops the latent codes from memorizing training data.
• Enforces Continuity: Encourages a smooth, organized latent space where interpolation is meaningful.
• Enables Generation: New samples can be generated by sampling z from the prior p(z) and passing it through the decoder.

The posterior is central to the Evidence Lower Bound (ELBO), the VAE's training objective: ELBO = E[log p_θ(x|z)] - D_KL(q_φ(z|x) || p(z)). The first term is the reconstruction loss, dependent on the decoder. The second term is the KL divergence, which depends entirely on the encoder's posterior approximation. Maximizing the ELBO simultaneously pushes the posterior to:

1. Explain the observed data well (high likelihood via the decoder).
2. Conform to our prior beliefs (low KL divergence).

A well-structured posterior is linked to disentangled representations. In a disentangled latent space, single dimensions correspond to independent, semantically meaningful factors of variation (e.g., pose, lighting, object identity). Variants like the β-VAE increase the weight (β > 1) on the KL term, applying stronger pressure for the posterior to match an isotropic Gaussian prior. This often encourages the model to learn more factorized and interpretable posterior distributions, though it may trade off some reconstruction fidelity.

The prior distribution is the initial probability distribution assumed for the latent variables before observing any data. In a standard VAE, this is typically a simple, fixed distribution like a standard multivariate normal distribution (N(0, I)). It represents our initial beliefs about the latent space structure.

• Role in VAE: Serves as a regularization target. The KL divergence term in the ELBO penalizes the learned posterior for deviating from this prior, encouraging a well-structured, continuous latent space.
• Common Choice: Isotropic Gaussian, chosen for mathematical convenience and to promote smooth interpolation.

The Evidence Lower Bound (ELBO) is the variational objective function maximized during VAE training. It provides a tractable lower bound to the intractable log-likelihood of the data (the evidence).

• Mathematical Form: ELBO = 𝔼[log p(x|z)] - KL(q(z|x) || p(z))
• Two Components: The first term is the reconstruction loss, encouraging accurate data generation. The second term is the KL divergence, regularizing the approximate posterior (q(z|x)) towards the prior (p(z)). Maximizing the ELBO indirectly improves the approximation of the true posterior.

Variational Inference (VI) is the overarching Bayesian approximation framework used by VAEs. It transforms the problem of computing the true, intractable posterior into an optimization problem.

• Core Idea: Instead of computing the exact posterior p(z|x), VI seeks the best approximation q(z|x) from a chosen family of simpler distributions (e.g., Gaussians).
• Measure of Fit: "Best" is defined by minimizing the KL divergence KL(q(z|x) || p(z|x)). This is equivalent to maximizing the ELBO.
• Amortization: In VAEs, this optimization is amortized by training a neural network (the encoder) to output the parameters of q(z|x) for any input x.

The probabilistic encoder, or inference network, is the neural network component that implements the approximate posterior distribution q(z|x). It maps input data x to the parameters (e.g., mean μ and log-variance log σ²) of a distribution in the latent space.

• Function: For an input image of a digit, the encoder doesn't output a single latent code, but a distribution (e.g., a Gaussian) from which a code is sampled.
• Output: Typically parameterizes a diagonal Gaussian distribution: q(z|x) = N(z; μ(x), σ²(x)I).
• Purpose: To perform fast, amortized inference, estimating the latent variables likely responsible for generating the observed data.

Kullback-Leibler (KL) Divergence is a non-symmetric measure of how one probability distribution diverges from a second, reference distribution. In the VAE objective, it acts as a regularizer.

• In the ELBO: The term KL(q(z|x) || p(z)) measures the divergence of the approximate posterior from the prior.
• Effect: Minimizing this term pushes the learned latent distributions to be close to the simple prior (e.g., standard normal). This prevents overfitting to the training data and encourages a smooth, organized latent space where interpolation is meaningful.
• Trade-off: It balances with the reconstruction loss; too much weight on the KL term can lead to posterior collapse.

Amortized inference is the efficiency technique central to VAEs. Instead of optimizing latent variables separately for each data point (which is computationally prohibitive), a shared neural network (the encoder) is trained to perform inference for any input.

• Analogy: Like amortizing a cost over time, the upfront cost of training the encoder is high, but after training, inference for a new data point is a single, cheap forward pass.
• Contrast with Traditional VI: Classic variational inference would optimize variational parameters for each datapoint individually. Amortized VI learns a global function (the encoder) that maps any x to its variational parameters.
• Benefit: Enables scalable application of Bayesian inference to large datasets.