Hierarchical VAE

A Hierarchical VAE structures its latent space across multiple stochastic layers, often arranged in a top-down generative process. Unlike a standard VAE with a single latent vector z, a hierarchical model uses a sequence z_1, z_2, ..., z_L. The top layer z_1 captures global, high-level attributes (e.g., object category), while lower layers z_L model local, fine-grained details (e.g., texture). This mirrors the hierarchical feature abstraction found in deep convolutional networks, allowing for more expressive and structured generation.

• Top-Down Generation: The prior is typically p(z_1)p(z_2|z_1)...p(z_L|z_{<L}), where each layer conditions on the ones above it.
• Bottom-Up Inference: The approximate posterior q(z|x) factorizes as q(z_L|x)q(z_{L-1}|z_L, x)..., enabling efficient inference of all latent variables from the data.

The layered latent structure is exceptionally powerful for modeling data with inherent multi-scale dependencies, such as video sequences and high-resolution images. For video, higher latent layers can represent the overall scene and action, while lower layers model frame-to-frame motion and pixel-level details. For images, the hierarchy can separate scene composition from object shapes and finally surface textures. This leads to generated samples that are globally consistent and locally plausible, addressing a key weakness of simpler generative models where long-range coherence often breaks down.

Posterior collapse is a common issue in deep VAEs where powerful decoders ignore latent variables, causing the KL divergence to vanish. The hierarchical architecture provides a natural defense. Lower latent layers, which are responsible for fine details, remain tightly coupled to the input data through the inference network, preventing them from being ignored. Meanwhile, higher layers can specialize in capturing more abstract statistics. This structural separation of information helps maintain an informative latent space across all layers, leading to better use of model capacity and improved learned representations.

While standard VAEs use a simple isotropic Gaussian prior N(0, I), hierarchical VAEs employ a structured prior with dependencies between layers (e.g., p(z_l | z_{<l})). This allows the model to learn complex, multi-modal distributions in the latent space that better match the aggregated posterior. This structure can also encourage disentanglement, where different layers—or groups of dimensions within a layer—captulate semantically distinct factors of variation. For instance, in face generation, one layer may control pose, another lighting, and another identity, enabling more precise and interpretable control over generation.

Despite the complexity of multiple latent layers, hierarchical VAEs maintain the amortized inference property of standard VAEs. A single, multi-branch inference network q(z|x) approximates the posterior for all latent variables simultaneously, avoiding the need for expensive per-datapoint optimization. Modern implementations use techniques like bottom-up inference networks paired with top-down generative models to share computational pathways efficiently. This makes training scalable to large datasets and deep hierarchies, which is critical for applications like generating high-resolution synthetic imagery or complex time-series data.

Hierarchical VAEs share a deep conceptual link with diffusion models. Both frameworks can be viewed as latent variable models with a Markov chain structure across many layers. In a hierarchical VAE, the chain goes from data x to high-level latents z_1 (inference) and back (generation). In diffusion, the forward process adds noise across many steps, and the reverse process denoises. This similarity has led to hybrid models and unified theoretical perspectives, showing that hierarchical latent variable models are a foundational architecture for state-of-the-art generative modeling, capable of achieving high log-likelihoods and sample quality.

A variational autoencoder (VAE) is the foundational deep generative model upon which hierarchical VAEs are built. It learns a probabilistic mapping between high-dimensional data and a lower-dimensional latent space by maximizing the evidence lower bound (ELBO). The ELBO balances reconstruction loss (data fidelity) and KL divergence (latent space regularization). Unlike a standard autoencoder, a VAE's encoder outputs parameters of a probability distribution, enabling structured sampling and generation.

A latent variable model is a statistical model that posits that observed data is generated from some underlying, unobserved (latent) variables. Hierarchical VAEs are a specific type of deep latent variable model where neural networks parameterize the complex relationships between layers of latent variables and the data. The hierarchy allows the model to capture multi-scale features, from broad outlines to fine details, in a structured probabilistic framework.

Variational inference (VI) is the core Bayesian approximation method used to train VAEs. It turns the intractable problem of computing the true posterior distribution into an optimization problem by finding the best approximation from a tractable family of distributions. Amortized inference, where a neural network (the encoder) learns to perform this approximation for any input, is what makes VAEs scalable. Hierarchical VAEs extend this by performing variational inference over a chain of latent variables.

A deep generative model uses deep neural networks to learn and sample from the underlying probability distribution of complex data, such as images, audio, or text. Hierarchical VAEs belong to this family, alongside Generative Adversarial Networks (GANs) and Diffusion Models. Their key distinction is the explicit, structured latent space and the principled probabilistic training via the ELBO. The hierarchical depth allows them to model data with intricate, multi-level dependencies more effectively than shallow generative models.

In the context of hierarchical VAEs, these are the key probability distributions governing the latent variables:

• Prior Distribution: The assumed probability distribution of latent variables before seeing data. In a hierarchical VAE, each layer of latent variables has its own prior, often conditioned on the layer above.
• Posterior Distribution: The conditional distribution of latent variables given the observed data. The probabilistic encoder learns to approximate this complex, true posterior. The hierarchy creates a structured posterior where higher-level latents capture abstract features and lower-level latents capture finer details.

A disentangled representation is a latent space where distinct, semantically meaningful factors of variation in the data are encoded in separate and statistically independent dimensions. While not an explicit goal of all hierarchical VAEs, the structured, multi-layer latent space can naturally encourage disentanglement. Techniques like β-VAE, which weights the KL divergence term in the ELBO more heavily, are often used to promote disentanglement, making the model's generative process more interpretable and controllable.