Variance-Preserving Process

The defining property of a VP process is that the total variance of the noisy sample (\mathbf{x}_t) is constrained to remain constant (typically 1) for all timesteps (t). This is enforced by ensuring the sum of the signal variance (from the original data) and the noise variance equals 1. Formally, if (\mathbf{x}_0) is the original data and (\epsilon) is standard Gaussian noise, the forward process is defined as:

[\mathbf{x}_t = \sqrt{\bar{\alpha}_t} \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon]

where the coefficients satisfy (\sqrt{\bar{\alpha}_t}^2 + (\sqrt{1 - \bar{\alpha}_t})^2 = 1). This ensures (\text{Var}(\mathbf{x}_t) = 1) if (\text{Var}(\mathbf{x}_0) = 1).

The process is governed by a noise schedule defined by sequences (\beta_t) and (\bar{\alpha}_t).

• (\beta_t): The variance of the noise added at step (t).
• (\bar{\alpha}_t): The cumulative product of the signal retention factors, defined as (\bar{\alpha}t = \prod{s=1}^{t} (1 - \beta_s)).

The VP constraint directly links these: (\bar{\alpha}_t + (1 - \bar{\alpha}_t) = 1), which is inherently true. The critical design choice is that (\beta_t) is scheduled such that (\bar{\alpha}_T \approx 0) at the final timestep (T), transforming the data into nearly pure Gaussian noise while preserving total variance throughout.

The VP process is one of two main parameterizations for diffusion, the other being the Variance-Exploding (VE) process.

Key Differences:

• VP: Total variance is constant. Noise is added, but signal is proportionally decayed.
• VE: Signal is kept constant ((\mathbf{x}_t = \mathbf{x}_0 + \sigma_t \epsilon)), allowing total variance to explode to infinity as (t) increases.

Practical Implication: VP processes are more common in image generation (e.g., DDPM) because they operate with normalized data and bounded variance, which can be more numerically stable. VE processes are often used in continuous-time formulations via Stochastic Differential Equations (SDEs).

The constant variance property leads to a simplified and stable training loss. The model (often a U-Net) is typically trained as a noise prediction network. Given a noisy sample (\mathbf{x}t), the model (\epsilon\theta) is trained to predict the noise component (\epsilon) that was added.

The simplified mean-squared error loss is: [L_\text{simple} = \mathbb{E}_{t, \mathbf{x}0, \epsilon} \left[ | \epsilon - \epsilon\theta(\mathbf{x}_t, t) |^2 \right]] This objective is equivalent to learning the score function (gradient of the log data density) under the VP parameterization, but the noise prediction form is often more straightforward to optimize.

To generate new data, the model reverses the VP process through iterative denoising. Starting from pure noise (\mathbf{x}_T \sim \mathcal{N}(0, I)), the reverse step is:

[\mathbf{x}_{t-1} = \frac{1}{\sqrt{1 - \beta_t}} \left( \mathbf{x}_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}t}} \epsilon\theta(\mathbf{x}_t, t) \right) + \sigma_t \mathbf{z}] where (\mathbf{z} \sim \mathcal{N}(0, I)) and (\sigma_t) is the reverse process variance. For DDPM, this is set to (\tilde{\beta}t = \frac{1 - \bar{\alpha}{t-1}}{1 - \bar{\alpha}_t} \beta_t). This iterative procedure gradually removes predicted noise, reconstructing a data sample from the learned distribution.

The VP process is the standard forward process for the foundational DDPM architecture. Its characteristics directly enable DDPM's reliable training and high-quality generation.

Why DDPM uses VP:

• Normalized Data: Assumes input data is scaled to have unit variance (e.g., images in [-1, 1]).
• Stable Training: The bounded variance and simple noise-prediction loss prevent gradient issues.
• Progressive Transformation: Provides a clear, smooth path from data to a tractable noise distribution (standard Gaussian). This parameterization was instrumental in demonstrating the practical potential of diffusion models for high-fidelity image synthesis.

The fixed Markov chain in a diffusion model that gradually adds Gaussian noise to a data sample over a series of timesteps, transforming it into pure noise. It is defined by a noise schedule. In a Variance-Preserving Process, this schedule is specifically designed so the total variance of the noisy sample remains constant (e.g., at 1) across all timesteps, unlike a Variance-Exploding process where variance increases.

A predefined function that controls the amount of variance (beta_t) of the Gaussian noise added at each timestep during the forward diffusion process. It determines the progression from data to noise.

• Key Parameter: The schedule of β_t values, typically increasing from near 0 to near 1.
• Variance-Preserving Connection: For VP processes, the schedule is constrained such that the product of all (1 - β_t) terms results in a final variance of 1. This is often implemented using a continuous function like β_t = 0.1 + (19.9 * t) for t in [0, 1] in some formulations.

A foundational class of generative model that learns to reverse a fixed forward process of gradually adding Gaussian noise to data. DDPMs popularized the noise prediction parameterization and the use of a simplified training objective. The original DDPM formulation uses a forward process that is a type of Variance-Preserving Process, where the variance of the noise is carefully scaled to ensure the final sample is approximately unit-variance Gaussian noise.

A training objective for generative models where a neural network (a score network) is trained to estimate the gradient (score) of the log probability density of the data distribution. In the context of diffusion models, the score function is proportional to the negative of the noise to be removed. Training a model via noise prediction (as in DDPM) is a practical, weighted form of score matching. The Variance-Preserving Process defines a specific continuous-time SDE whose score function the model learns to approximate.

A continuous-time framework for modeling the forward and reverse processes in diffusion models, where data corruption and generation are described by differential equations with a stochastic (noise) component.

• Variance-Preserving SDE: Corresponds to the Variance Preserving (VP) SDE, often written as dx = -0.5 β(t) x dt + sqrt(β(t)) dw, where β(t) is the noise schedule. This SDE ensures the marginal variance of x(t) remains constant.
• Alternative: The Variance Exploding (VE) SDE, where variance grows unbounded.

An ordinary differential equation (ODE) derived from the reverse-time stochastic differential equation (SDE) of a diffusion model. It describes a deterministic sampling trajectory that shares the same marginal probability distributions as the stochastic reverse process. For every diffusion process (including the Variance-Preserving SDE), there exists a corresponding Probability Flow ODE. This enables the use of fast, deterministic ODE solvers for sampling, trading off some potential diversity for faster and more consistent generation.