Prior Distribution

A prior distribution formalizes domain knowledge or assumptions about latent variables before any data is seen. In a standard Variational Autoencoder (VAE), the prior is typically a standard normal distribution, N(0, I), representing a neutral assumption of independent, unit-variance latent factors. This choice is mathematically convenient but can be replaced with more informative priors to inject specific structural knowledge, such as sparsity or hierarchical relationships.

The prior acts as a regularizer during training. In the VAE objective, the Kullback-Leibler (KL) divergence term penalizes the learned posterior distribution for deviating from the prior. This prevents overfitting to the training data by encouraging the latent codes to conform to a simple, well-behaved distribution. A stronger prior (e.g., with lower variance) imposes tighter constraints, often leading to a more compact and disentangled latent representation.

The prior is the foundational component of Bayes' theorem, which updates beliefs to form the posterior distribution after observing data: Posterior ∝ Likelihood × Prior. In VAEs, the true posterior is intractable, so the model uses amortized variational inference to approximate it. The quality of this approximation is directly influenced by the choice of prior, as it defines the target for the variational posterior.

For generative modeling, the prior defines the distribution from which latent vectors are sampled during synthesis. To generate new data, one samples a vector z ~ p(z) from the prior and passes it through the probabilistic decoder. The properties of the prior therefore control the diversity and characteristics of the generated outputs. A poorly chosen prior can lead to low-quality or low-diversity samples.

• Uninformative/Weak Prior: Makes minimal assumptions (e.g., a broad Gaussian). Used when domain knowledge is limited.
• Conjugate Prior: Chosen so the posterior belongs to the same family as the prior, simplifying mathematical analysis (e.g., Gaussian prior for Gaussian likelihood).
• Hierarchical Prior: A prior whose parameters are themselves drawn from a distribution, adding flexibility to model complex data.
• Sparsity-Inducing Prior: Encourages many latent dimensions to be zero (e.g., Laplace prior). Useful for feature selection.
• Empirical Prior: Informed by previous data or experiments, moving beyond purely theoretical assumptions.

The choice of prior has direct, measurable effects:

• Posterior Collapse: An overly flexible decoder paired with a weak prior can cause the KL term to vanish, making the latent space uninformative.
• Disentanglement: Priors like the one used in β-VAE (with an adjusted weight β) can promote learning of independent latent factors.
• Computational Tractability: Simple priors (like Gaussians) enable the use of the reparameterization trick, allowing efficient gradient-based learning. Complex priors may require more advanced inference techniques.

The posterior distribution is the updated probability distribution of the latent variables after observing the data. It represents the model's beliefs about the latent factors that generated the observed input. In a VAE, the probabilistic encoder (inference network) approximates this intractable true posterior with a variational distribution, typically a Gaussian parameterized by a neural network.

• Bayes' Rule: Calculated as Posterior ∝ Likelihood × Prior.
• Variational Approximation: The encoder outputs parameters (mean, variance) defining an approximate posterior q(z|x).
• Inference Goal: The aim of training is to make this approximate posterior as close as possible to the true posterior.

The Evidence Lower Bound (ELBO) is the objective function maximized during VAE training. It provides a tractable lower bound to the log-likelihood (evidence) of the data. The ELBO explicitly contains the prior distribution term.

• Mathematical Form: ELBO = 𝔼[log p(x|z)] - KL( q(z|x) || p(z) ).
• Two Components: The first term is the reconstruction loss. The second term is the KL divergence between the approximate posterior and the prior distribution.
• Role of the Prior: The KL term acts as a regularizer, pushing the learned latent distribution q(z|x) to match the chosen prior p(z), preventing overfitting and structuring the latent space.

Kullback-Leibler (KL) Divergence is a non-symmetric measure of how one probability distribution diverges from a second, reference distribution. In the VAE objective, it quantifies the difference between the encoder's output distribution and the prior.

• In the ELBO: The term KL( q(z|x) || p(z) ) measures the extra information required to represent the latent code using the prior p(z) versus using the posterior approximation q(z|x).
• Regularization Effect: Minimizing this divergence forces the aggregated posterior (over all data) to resemble the prior, ensuring the latent space is well-structured (e.g., a smooth, continuous manifold for a Normal prior).
• Vanishing KL: If this term collapses to zero (posterior collapse), the latent variable is ignored, and the model fails to learn a useful representation.

Variational Inference (VI) is the overarching Bayesian approximation framework in which VAEs operate. It transforms the problem of computing an intractable posterior into an optimization problem.

• Core Idea: Instead of computing the true posterior p(z|x), VI posits a family of simpler distributions Q and finds the member q*(z) ∈ Q that is closest to the true posterior (minimizing KL divergence).
• Amortized VI: A VAE uses amortized inference, where a single neural network (the encoder) learns to output the optimal q(z) parameters for any input x, rather than optimizing per datapoint.
• Prior's Role: The prior p(z) defines the reference distribution for the latent variables within the probabilistic model being approximated.

The latent space is the lower-dimensional, compressed representation where the model captures the essential factors of data variation. The prior distribution defines the expected statistical properties of this space.

• Structure & Continuity: A standard Normal prior (p(z) = N(0,I)) encourages the latent space to be a continuous, smooth, and centered manifold. This enables meaningful interpolation and random sampling.
• Sampling for Generation: To generate new data, one samples a latent vector z ~ p(z) from the prior and passes it through the decoder.
• Disentanglement: Certain priors or objective modifications (like in β-VAE) can encourage a disentangled representation, where single latent dimensions correspond to independent, interpretable data attributes.

Mean-field approximation is a specific, simplifying assumption commonly used in variational inference, including in the default VAE encoder. It assumes all latent dimensions are mutually independent.

• Factorization: The variational posterior is factorized: q(z|x) = ∏ᵢ q(zᵢ|x).
• Encoder Output: In practice, the VAE encoder outputs a diagonal covariance matrix for a Gaussian posterior, meaning no covariance (dependence) between latent dimensions is modeled.
• Interaction with Prior: The standard Normal prior is also fully factorized. The mean-field assumption simplifies the calculation of the KL divergence in the ELBO to a sum of KL terms for each dimension.