Denoising Diffusion Probabilistic Model (DDPM)

A fixed Markov chain that gradually destroys the data structure by adding small amounts of Gaussian noise over T timesteps. At each step t, the image becomes progressively noisier until, at the final step, the data distribution converges to an isotropic Gaussian. This process requires no learning and is defined by a variance schedule β₁, ..., βₜ that controls the noise injection rate.

• The forward process posterior q(xₜ|xₜ₋₁) is a simple Gaussian
• Allows direct sampling of noisy latents at arbitrary timesteps via the reparameterization trick
• High β values near the end ensure the terminal distribution is pure noise

A learned Markov chain that iteratively removes noise, starting from pure Gaussian noise xₜ ~ N(0, I) and progressing backward to reconstruct a clean sample x₀. A neural network ε_θ(xₜ, t) predicts the noise component at each step, enabling the model to denoise the latent representation.

• Parameterized by a U-Net with self-attention layers
• Each reverse step samples from p_θ(xₜ₋₁|xₜ)
• The model learns to estimate the score function ∇ₓ log p(x) of the data distribution

The training objective simplifies to a weighted variational bound that reduces to predicting the noise ε added at each timestep. The network minimizes the mean squared error between the true noise and its prediction: L_simple = E[||ε - ε_θ(xₜ, t)||²].

• Avoids directly modeling the intractable marginal likelihood
• Connection to denoising score matching with Langevin dynamics
• Enables stable training without adversarial objectives or mode collapse

A technique that trades off sample diversity for fidelity by jointly training a conditional and unconditional diffusion model. During sampling, the noise prediction is extrapolated away from the unconditional prediction: ε̂_θ = (1 + w)ε_θ(xₜ, c, t) - w ε_θ(xₜ, t), where w is the guidance scale.

• Eliminates the need for a separate classifier network
• Higher w values produce more faithful but less diverse samples
• Critical for text-to-image models like Stable Diffusion and DALL·E 2

A method to accelerate the slow iterative sampling process by training a student model to halve the number of required steps. The student learns to predict the output of two teacher denoising steps in a single forward pass, effectively compressing the sampling trajectory.

• Reduces sampling from 1000+ steps to as few as 4-8 steps
• Preserves sample quality while dramatically lowering inference latency
• Enables real-time generation on consumer hardware

Instead of operating directly in pixel space, the diffusion process occurs in a compressed latent space learned by a pretrained autoencoder. The encoder E maps images to latents z = E(x), diffusion operates on z, and the decoder D reconstructs the final image.

• Reduces computational cost by orders of magnitude
• Perceptual compression retains semantic content while discarding imperceptible details
• Foundation of Stable Diffusion and other efficient generative models

A deep learning architecture where two neural networks—a generator and a discriminator—compete adversarially. The generator creates synthetic samples, while the discriminator attempts to distinguish them from real data. Unlike DDPMs, which use a fixed Markov chain of noise addition and removal, GANs rely on this minimax game to achieve realism. Key distinctions from diffusion models include:

• Training instability due to mode collapse
• Single-pass generation (faster inference)
• Implicit density modeling rather than explicit likelihood maximization

A generative model that encodes input data into a probabilistic latent space and decodes samples from that space to generate new records. VAEs maximize a variational lower bound (ELBO) on the data likelihood. Compared to DDPMs:

• VAEs produce blurrier samples due to the Gaussian prior assumption
• DDPMs achieve higher fidelity through iterative refinement
• Both share the concept of learning to reverse a corruption process
• VAEs provide an explicit latent representation useful for interpolation

The degree to which a synthetic dataset preserves the univariate distributions, multivariate correlations, and aggregate statistics of the original real data. For DDPM-generated data, fidelity is measured through:

• Column shape comparisons using KS-complement metrics
• Pairwise correlation preservation scores
• Boundary adherence to ensure no impossible values
• Higher fidelity typically comes at the cost of reduced privacy guarantees, embodying the privacy-utility trade-off

A failure condition primarily observed in GAN training where the generator learns to produce only a limited variety of outputs, failing to capture the full diversity of the real data distribution. DDPMs inherently resist mode collapse because:

• They model the full data distribution through explicit likelihood training
• The diffusion process covers all modes of the target distribution
• Each denoising step is trained to recover any possible input
• This makes DDPMs particularly valuable for fairness-aware synthesis where demographic coverage is critical

The fundamental balancing act between the strength of a privacy guarantee and the statistical usefulness of synthetic data. DDPMs trained without explicit privacy mechanisms can memorize and reproduce training samples. Mitigation strategies include:

• Training with differential privacy via DP-SGD
• Early stopping before full memorization occurs
• Adding noise to the conditioning signal
• Evaluating re-identification risk with membership inference attacks The optimal operating point depends on the downstream use case and regulatory requirements.