Variational Autoencoder

Unlike standard autoencoders that map inputs to a single fixed point, the encoder in a VAE outputs the parameters of a probability distribution—typically a mean vector (μ) and a log-variance vector (log σ²). This forces the model to learn a smooth, continuous latent space where similar inputs map to overlapping regions. The encoder defines an approximate posterior distribution q(z|x) that is regularized to be close to a prior, usually a standard Gaussian p(z) = N(0, I). This probabilistic bottleneck is the core mechanism enabling generative sampling.

The critical innovation that makes VAEs trainable via backpropagation. Sampling z directly from N(μ, σ²) is a stochastic operation that blocks gradient flow. The reparameterization trick expresses the latent vector as a deterministic function: z = μ + σ ⊙ ε, where ε ~ N(0, I) is an auxiliary noise variable sampled independently. This shifts the stochasticity to the input ε, allowing gradients to flow through the network to update μ and σ while preserving the model's probabilistic nature.

VAEs are trained by maximizing the ELBO, a variational lower bound on the intractable marginal log-likelihood log p(x). The ELBO decomposes into two competing terms:

• Reconstruction Loss: Maximizes the expected log-likelihood E[log p(x|z)], ensuring the decoder accurately reconstructs inputs. For continuous data, this is often Mean Squared Error; for binary data, Binary Cross-Entropy.
• KL Divergence Regularizer: Minimizes D_KL(q(z|x) || p(z)), forcing the learned latent distribution to match the prior. This acts as a regularizer, preventing the variance from collapsing to zero and enforcing latent space continuity.

A well-trained VAE often exhibits disentanglement, where individual latent dimensions correspond to independent, semantically meaningful generative factors. The β-VAE variant strengthens this by adding a hyperparameter β > 1 to weight the KL divergence term, imposing a stronger information bottleneck. This pressure encourages the model to learn factorized representations where, for example, one dimension controls digit thickness and another controls rotation angle, enabling controlled generation and interpretable feature manipulation.

VAEs provide a principled method for anomaly detection using reconstruction probability, not raw reconstruction error. Instead of comparing input x to output x', the process samples z multiple times from the encoder's distribution and computes the mean probability E[p(x|z)]. Anomalous inputs, unseen during training, map to low-density regions of the latent space, yielding low reconstruction probability. This probabilistic metric accounts for the model's inherent uncertainty and provides a more robust anomaly score than deterministic autoencoders.

The enforced continuity of the latent space enables semantically smooth interpolation between data points. By encoding two inputs to their respective latent distributions, sampling points along a linear trajectory between z₁ and z₂, and decoding these intermediate vectors, the VAE generates a perceptually coherent morphing sequence. This capability is a direct consequence of the KL divergence regularizer, which prevents gaps in the latent manifold and ensures that every point in the high-density region decodes to a plausible, realistic output.