Generative Model

Variational Autoencoders (VAEs) are a class of generative models that learn a latent space representation of data by combining an encoder network (which maps data to a distribution in latent space) and a decoder network (which reconstructs data from latent points).

• Core Mechanism: They are trained by maximizing the Evidence Lower Bound (ELBO), which balances reconstruction accuracy with a regularization term (the Kullback-Leibler Divergence) that encourages the learned latent distribution to be smooth and continuous.
• Key Use Cases: Image generation, data compression, and learning disentangled representations where independent factors of variation (like object shape and color) are encoded in separate latent dimensions.
• Limitation: Generated samples can often be blurrier than those from other models due to the inherent approximations in variational inference.

Generative Adversarial Networks (GANs) employ a two-player minimax game between a generator network (which creates synthetic data) and a discriminator network (which tries to distinguish real from fake data).

• Core Mechanism: The generator aims to produce data so convincing that the discriminator cannot tell it apart from the real training data. This adversarial training drives the generator to learn the true data distribution.
• Key Use Cases: High-fidelity image and video synthesis, style transfer, and data augmentation. Pioneering models like StyleGAN demonstrated unprecedented control over attributes like facial features and artistic style.
• Challenges: Training instability (mode collapse, where the generator produces limited varieties of samples) and difficulty in convergence are well-known issues.

Autoregressive models generate data sequentially, predicting the next element in a sequence (e.g., a pixel or a word) based on all previously generated elements. They explicitly model the conditional probability distribution of the data.

• Core Mechanism: They factorize the joint probability of a sequence into a product of conditional probabilities: p(x) = p(x_1) * p(x_2 | x_1) * ... * p(x_n | x_1,...,x_{n-1}).
• Key Use Cases: Text generation (GPT models), waveform synthesis (WaveNet), and high-quality image generation (PixelCNN, Image GPT).
• Characteristics: These models are inherently likelihood-based, allowing for exact calculation of data probability. A primary drawback is their sequential nature, which makes generation slower than parallelizable models.

Normalizing Flows are a family of generative models that learn an invertible, differentiable transformation (a flow) between a simple base distribution (e.g., a Gaussian) and the complex target data distribution.

• Core Mechanism: By applying a sequence of bijective transformations, they can map data points to latent variables and back exactly. This allows for both efficient sampling and exact likelihood computation.
• Key Use Cases: Density estimation, data generation, and applications requiring exact probabilistic inference, such as variational inference with complex posterior distributions.
• Advantage: Unlike VAEs, they provide exact latent-variable inference and log-likelihoods. The requirement for invertibility often imposes architectural constraints that can limit model flexibility.

Energy-Based Models (EBMs) represent a probability distribution by associating an unnormalized scalar energy to each data configuration. Lower energy corresponds to more probable, plausible data.

• Core Mechanism: The probability is defined as p(x) = exp(-E(x)) / Z, where Z is the intractable normalizing constant (the partition function). Training involves contrasting the energies of real data (lowered) and generated samples (raised), often using techniques like contrastive divergence.
• Key Use Cases: They are a highly general framework used for tasks like anomaly detection, denoising, and as components in other models. Recent advancements have integrated them with other architectures for improved sample quality.
• Challenge: Sampling and computing the partition function Z are typically difficult, requiring approximate methods like Markov Chain Monte Carlo (MCMC).

Diffusion models (or denoising diffusion probabilistic models) generate data by progressively reversing a forward diffusion process that gradually adds noise to data until it becomes pure Gaussian noise.

• Core Mechanism: A neural network (the denoiser or score network) is trained to predict and remove the noise added at each step of the forward process. Generation involves iteratively denoising a sample of pure noise.
• Key Use Cases: State-of-the-art image, audio, and video generation (e.g., DALL-E 3, Stable Diffusion). They are known for producing highly diverse and high-fidelity samples.
• Advantages: More stable training than GANs and do not suffer from mode collapse. The iterative process, while computationally intensive during sampling, is highly parallelizable across the denoising steps.

A world model is an internal, learned representation within an AI system that captures the dynamics and regularities of its environment. It enables the agent to simulate and predict future states without direct interaction, forming the core of model-based planning and imagination. In reinforcement learning, a world model acts as a surrogate environment, allowing for cheap, internal rollouts to evaluate potential actions.

• Key Function: Enables counterfactual reasoning and planning by answering "what-if" questions.
• Architecture: Often implemented as a recurrent neural network or transformer that predicts the next latent state and reward.
• Example: An autonomous vehicle's world model predicts how traffic will evolve seconds ahead based on current sensor input.

A latent space is a lower-dimensional, continuous vector space where learned representations of data reside. Generative models learn to map high-dimensional data (like images or text) to points in this space, capturing the essential factors of variation. Operations within the latent space, such as interpolation between points, enable controlled generation of new, plausible data samples.

• Core Property: Encodes semantic meaning; similar data points cluster together.
• Use in Generation: Sampling a random vector from the latent space and passing it through a decoder network produces a novel data instance.
• Goal: A well-structured latent space facilitates tasks like style transfer, attribute manipulation, and anomaly detection.

A Variational Autoencoder (VAE) is a foundational generative model that learns to encode data into a regularized latent space and decode it back. It combines an encoder network (which maps data to a distribution in latent space) and a decoder network (which reconstructs data from latent points). Its training objective maximizes the Evidence Lower Bound (ELBO), which balances reconstruction accuracy with a regularization term (the KL divergence) that encourages a smooth, organized latent space.

• Key Innovation: Introduces stochasticity in the encoding process, enabling smooth interpolation and generation.
• Limitation: Tends to produce blurrier samples compared to later models like GANs or Diffusion Models.
• Application: Used for data compression, denoising, and as a component in more complex hierarchical models.

A diffusion model is a class of generative model that learns to reverse a gradual forward diffusion process, which systematically adds noise to data until it becomes pure Gaussian noise. The model is trained to denoise, learning a reverse diffusion process that iteratively transforms random noise into a coherent data sample. This process typically involves hundreds of steps, making it computationally intensive but capable of producing extremely high-fidelity and diverse outputs.

• Training: A neural network (often a U-Net) is trained to predict the noise added at each step of the forward process.
• Sampling: Generation is performed by starting with random noise and applying the learned denoising steps sequentially.
• Dominant Use Case: The state-of-the-art architecture for photorealistic image generation (e.g., DALL-E 3, Stable Diffusion).

A Generative Adversarial Network (GAN) is a generative model framework based on an adversarial game between two neural networks: a Generator (G) that creates synthetic data, and a Discriminator (D) that tries to distinguish real data from fakes. They are trained simultaneously in a minimax game; the generator improves its outputs to fool the discriminator, while the discriminator becomes better at detection.

• Core Objective: The generator learns to produce data that lies on the true data manifold.
• Strengths: Can generate very sharp, high-quality samples (especially images).
• Challenges: Prone to training instability and mode collapse, where the generator produces limited varieties of samples.

A normalizing flow is a generative model that constructs a bijective (invertible) mapping between a complex data distribution and a simple base distribution (like a Gaussian). It transforms a sample from the simple distribution through a sequence of invertible neural network layers to produce a complex data sample. The key advantage is the ability to compute the exact likelihood of any generated data point, which is valuable for density estimation.

• Mathematical Basis: Uses the change-of-variables formula to compute probabilities.
• Property: Because the mapping is invertible, you can also encode any data point back into its latent representation.
• Trade-off: The requirement for invertibility often constrains model architecture, making flows less flexible than VAEs or GANs for some data types.