Diffusion Model

The model's operation is defined by two Markov chains. The forward process is a fixed procedure that gradually adds Gaussian noise to a data sample over T timesteps until it becomes pure noise. The reverse process is a learned neural network that predicts how to denoise a sample, moving backwards from timestep T to 0 to generate new data. This iterative denoising is the core generative act.

Instead of predicting the noise directly, many diffusion models are trained using score matching. The model learns to estimate the score function—the gradient of the log probability density of the data—at each noise level. By following these score estimates (steering towards higher data density), the model can reverse the diffusion process. This is mathematically equivalent to predicting the noise component of a noisy sample.

To reduce computational cost, many practical diffusion models (e.g., Stable Diffusion) operate in a compressed latent space, not pixel space. An encoder (like a VAE) first compresses an image into a lower-dimensional latent representation. The diffusion process denoises within this latent space, and a decoder then reconstructs the final high-resolution image. This enables high-quality synthesis with significantly less memory and compute.

Diffusion models excel at conditional generation, where synthesis is guided by an external input. Common conditions include:

• Text prompts (via cross-attention layers in text-to-image models)
• Class labels
• Spatial maps (e.g., depth, edges, pose via architectures like ControlNet) Guidance strength is controlled by a guidance scale parameter, which amplifies the influence of the condition versus unconditional generation, trading off sample fidelity for diversity.

The reverse process can be executed with different samplers, offering a trade-off between speed and quality.

• Stochastic samplers (e.g., DDPM) introduce noise during each denoising step, often yielding higher-quality results but requiring many steps (e.g., 1000).
• Deterministic samplers (e.g., DDIM) formulate the reverse process as an ODE, allowing for fewer, noise-free steps and enabling meaningful interpolation in latent space. Sampler choice is a key engineering decision for deployment.

Vs. GANs: Diffusion models avoid mode collapse and training instability common in adversarial setups. They provide better coverage of the data distribution but are typically slower at inference due to iterative steps. Vs. VAEs: Diffusion models do not rely on a potentially restrictive prior (like a standard Gaussian) and generally produce higher-fidelity samples, as they learn to model the data distribution more precisely through the gradual denoising process.

A Generative Adversarial Network (GAN) is a generative model architecture based on an adversarial game between two neural networks: a generator that creates data and a discriminator that tries to distinguish real data from fakes. This minimax competition drives the generator to produce increasingly realistic samples. GANs were the dominant paradigm for image synthesis before the rise of diffusion models.

• Core Mechanism: Adversarial loss from the discriminator network.
• Challenges: Mode collapse and unstable training dynamics.
• Common Variants: StyleGAN, CycleGAN, and Progressive GANs.

A Variational Autoencoder (VAE) is a probabilistic generative model that learns a compressed, structured latent representation of input data. It consists of an encoder that maps data to a distribution in latent space and a decoder that reconstructs data from samples of that distribution. VAEs optimize the Evidence Lower Bound (ELBO), balancing reconstruction accuracy with latent space regularity.

• Key Feature: Provides a continuous, interpolatable latent space.
• Role in Diffusion: Often used in latent diffusion models (like Stable Diffusion) as the perceptual compressor.
• Limitation: Can produce blurrier samples compared to GANs or diffusion models.

A Neural Radiance Field (NeRF) is a deep learning model that represents a 3D scene as a continuous volumetric function. Given a set of 2D images with known camera poses, a NeRF model learns to map a 3D spatial coordinate and viewing direction to a volume density and view-dependent color. This enables high-fidelity novel view synthesis.

• Output: Renders new views via volume rendering (ray marching).
• Application in Synthetic Data: Generating perfectly labeled, multi-view consistent 3D training data for perception tasks.
• Relation to Diffusion: Diffusion models are increasingly used to generate or refine NeRF scenes from sparse inputs.

Physically Based Rendering (PBR) is a computer graphics methodology that uses real-world physics principles to simulate light interaction with materials. It relies on measured material properties and the rendering equation to produce photorealistic images. PBR is a cornerstone of modern simulation engines for generating high-fidelity synthetic data.

• Core Components: Bidirectional Reflectance Distribution Function (BRDF), accurate lighting models, and HDR environments.
• Key Benefit: Generates images with consistent material appearance under any lighting condition.
• Use Case: Creating ultra-realistic synthetic scenes for training vision models where visual fidelity is critical.

The Fréchet Inception Distance (FID) is the standard metric for evaluating the quality and diversity of images generated by models like GANs and diffusion models. It compares the statistics of feature embeddings from a pre-trained Inception-v3 network for both the real and generated image sets. A lower FID score indicates that the two sets are more similar.

• Calculation: Uses the mean and covariance of the embeddings in the feature space.
• Purpose: Provides a single, quantitative score to benchmark generative model performance.
• Limitation: Relies on features relevant for ImageNet classification, which may not align with all downstream tasks.