Denoising Diffusion Probabilistic Model (DDPM)

The forward process is a fixed Markov chain that gradually adds Gaussian noise to a data sample over a series of T timesteps. It is defined by a noise schedule (beta_t), which controls the variance of the noise added at each step. This process has no learnable parameters and transforms any data point x_0 into pure noise x_T, following a predefined distribution (e.g., an isotropic Gaussian).

• Mathematical Definition: q(x_t | x_{t-1}) = N(x_t; sqrt(1 - beta_t) * x_{t-1}, beta_t * I)
• Purpose: Provides a structured path of corrupted data for the model to learn to reverse.

The reverse process is the learned generative trajectory. Starting from pure noise x_T ~ N(0, I), a neural network parameterizes a series of denoising steps to progressively remove noise, ultimately synthesizing a new data sample x_0. This process approximates the reversal of the forward Markov chain.

• Mathematical Definition: p_θ(x_{t-1} | x_t) = N(x_{t-1}; μ_θ(x_t, t), Σ_θ(x_t, t))
• Core Task: The model learns to predict the mean μ_θ and often the variance Σ_θ of the Gaussian distribution for the previous, less noisy step.

This is the core trainable neural network, typically a U-Net architecture. Instead of directly predicting the denoised sample x_0, it is commonly parameterized to predict the noise component ε added to the noisy input x_t at timestep t. This noise prediction objective is simple and stable for training.

• Architecture Features: Uses an encoder-decoder with skip connections to capture multi-scale spatial features. Incorporates timestep embeddings to condition the network on the current diffusion step.
• Equivalence: Predicting the noise ε_θ(x_t, t) is functionally equivalent to learning the score function ∇ log p(x_t).

The noise schedule is a critical, predefined function that determines the progression of the forward process. It defines the variance hyperparameters {β_1, β_2, ..., β_T}, where β_t ∈ (0,1). Common schedules are linear, cosine, or sigmoid.

• Variance-Preserving Process: A common parameterization where α_t = 1 - β_t and ᾱ_t = Π_{s=1}^t α_s. This ensures the variance of x_t remains bounded.
• Impact: The schedule controls the trade-off between perceptual quality and sampling speed. It dictates how much signal is destroyed at each step, influencing the difficulty of the denoising task for the network.

DDPMs are trained by optimizing a variational bound on the negative log likelihood. The practical, simplified objective is a mean-squared error loss between the true noise added during the forward process and the noise predicted by the network.

• Loss Function: L_simple = E_{x_0, ε, t}[ || ε - ε_θ(x_t, t) ||^2 ] Where ε is the true noise sampled from N(0, I), t is uniformly sampled from {1,...,T}, and x_t = sqrt(ᾱ_t) * x_0 + sqrt(1 - ᾱ_t) * ε.
• Interpretation: This objective directly trains the U-Net to perform one-step denoising at any arbitrary noise level t.

Sampling is the iterative application of the learned reverse process. Starting from random noise x_T, the algorithm uses the trained network's predictions to gradually denoise the sample over T steps.

• Ancestral Sampling: The standard method for DDPMs. For each step from t=T to 1:
  1. Predict the noise: ε_θ = model(x_t, t)
  2. Compute the mean for x_{t-1}: μ_θ(x_t, t)
  3. Sample: x_{t-1} = μ_θ + σ_t * z, where z ~ N(0, I) and σ_t is derived from the schedule.
• Deterministic Variants: Methods like DDIM formulate a non-Markovian process that allows for faster sampling with fewer steps by making the process deterministic.

The fixed Markov chain that gradually adds Gaussian noise to a clean data sample over a series of timesteps, transforming it into pure noise. This is a predefined, non-learned corruption process.

• Key Parameter: The noise schedule (beta_t) controls the variance of the noise added at each step.
• Mathematically, given data (x_0), the noisy sample at step (t) is (x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon), where (\epsilon) is standard Gaussian noise and (\bar{\alpha}_t) is a product of noise schedule terms.
• The process is designed so that (x_T) approximates an isotropic Gaussian distribution.

The learned generative trajectory that iteratively denoises a sample of pure noise, reversing the forward process to synthesize new data from the target distribution. This is the core of the DDPM's generative capability.

• A neural network (e.g., a U-Net) is trained to predict the noise (\epsilon) in a noisy sample (x_t).
• Starting from noise (x_T \sim \mathcal{N}(0, I)), the model applies a series of denoising steps: (x_{t-1} = \frac{1}{\sqrt{\alpha_t}} (x_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}t}} \epsilon\theta(x_t, t)) + \sigma_t z), where (z) is new noise.
• This defines a parameterized Markov chain that gradually removes noise.

A predefined function that controls the amount of variance (beta_t) of the Gaussian noise added at each timestep during the forward diffusion process. It determines the progression from data to pure noise.

• Common schedules are linear, cosine, or sigmoid functions mapping timestep (t) to variance (\beta_t).
• The schedule is critical for training stability and sampling quality. A variance-preserving schedule ensures the total variance of the noisy sample remains constant (e.g., 1).
• The cumulative product (\bar{\alpha}t = \prod{s=1}^{t} (1 - \beta_s)) defines how much signal from the original data (x_0) remains at step (t).

The most common parameterization for the neural network in a DDPM, where the model is trained to predict the noise component (\epsilon) that was added to a clean sample to produce a noisy input (x_t) at a given timestep (t).

• This objective is equivalent to score matching, as predicting the noise is proportional to learning the gradient (score) of the log data density.
• The training loss is a simple mean-squared error: (L = \mathbb{E}{t, x_0, \epsilon} [| \epsilon - \epsilon\theta(\sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon, t) |^2]).
• This parameterization leads to stable training and is used in foundational models like the original DDPM.

A convolutional neural network architecture characterized by a symmetric encoder-decoder structure with skip connections, commonly used as the backbone for the noise prediction network in image-based diffusion models.

• The encoder downsamples the spatial resolution to capture contextual information.
• The decoder upsamples to reconstruct the image detail.
• Skip connections between corresponding encoder and decoder layers preserve fine-grained spatial information lost during downsampling.
• The network is typically conditioned on the timestep (t) via sinusoidal embeddings or adaptive group normalization layers.

The standard stochastic sampling procedure for DDPMs, where each denoising step in the reverse process involves adding a small, fresh amount of Gaussian noise.

• This procedure follows the reverse Markov chain defined by the model's learned (or fixed) variances (\sigma_t^2).
• The injection of new noise at each step makes the generation process stochastic, allowing for diverse outputs from the same initial noise.
• It contrasts with deterministic samplers like DDIM, which define a non-Markovian forward process to enable faster, noise-free sampling.