Stochastic Differential Equation (SDE)

The drift coefficient, often denoted as f(x_t, t), is the deterministic component of the SDE. It defines the predictable, average direction of change for the data variable x_t over an infinitesimal time step.

• In the forward process, the drift is typically designed to push the data towards zero or a prior distribution.
• In the reverse process, the learned drift guides the denoising trajectory back towards the data manifold.
• It is analogous to the mean of the transition kernel in discrete-time diffusion models.

The diffusion coefficient, denoted as g(t), controls the magnitude of the stochastic (noise) term in the SDE. It is a scalar function of time that determines how much random noise is injected at each instant.

• It governs the variance schedule of the corruption process.
• A common choice is g(t) = √(dβ/dt), linking it to the noise schedule β(t) from discrete models.
• In the variance-exploding SDE formulation, g(t) increases over time, while in the variance-preserving SDE, it is coupled with the drift to keep total variance constant.

The Wiener process, W_t (or dB_t), is the source of continuous random noise in the SDE. Its infinitesimal increments dW_t are normally distributed with mean zero and variance dt.

• It provides the fundamental stochastic driver for the forward corruption: dx = f(x,t)dt + g(t)dW.
• The reverse-time SDE, used for generation, involves a backward Wiener process and a correction term dependent on the score function.
• This mathematical object formalizes the continuous addition of Gaussian noise, generalizing the discrete noise addition steps in models like DDPM.

The score function, ∇_x log p_t(x), is the gradient of the log-probability of the data distribution at time t. It is the critical learned quantity that steers the reverse SDE.

• The reverse-time SDE for generation is: dx = [f(x,t) - g(t)² ∇_x log p_t(x)] dt + g(t) dW̄.
• The term -g(t)² ∇_x log p_t(x) is a drift correction that points towards regions of high data density.
• A neural score network s_θ(x_t, t) is trained to approximate this function, typically via score matching or as a noise prediction network.

The Probability Flow Ordinary Differential Equation (ODE) is a deterministic counterpart to the reverse-time SDE that shares the same marginal probability distributions p_t(x) at every time t.

• It is derived by removing the stochastic noise term from the reverse SDE: dx = [f(x,t) - ½ g(t)² ∇_x log p_t(x)] dt.
• This ODE defines a deterministic sampling trajectory, enabling faster samplers and exact likelihood computation.
• It connects diffusion models to continuous normalizing flows, where the learned score defines the flow's vector field.

The continuous time variable t ∈ [0, T] parameterizes the entire diffusion trajectory, replacing the discrete timestep index. Time flows from t=0 (data distribution) to t=T (approximate noise distribution).

• The drift f(x,t) and diffusion g(t) coefficients are explicitly functions of time.
• The score network s_θ(x_t, t) is conditioned on time, often via sinusoidal embeddings or learned embeddings.
• The choice of T and the functional forms of f and g define the specific SDE formulation (e.g., Variance Preserving, Variance Exploding).

The fixed Markov chain in a diffusion model that gradually adds Gaussian noise to a data sample over a series of timesteps, transforming it into pure noise. This is the corruption process modeled by the forward SDE.

• Mathematically, it's defined by a stochastic differential equation: dx = f(x, t)dt + g(t)dw, where dw is a Wiener process (Brownian motion).
• The noise schedule determines the coefficients f and g, controlling the rate of corruption.
• The process ends at a predefined terminal time T, where the data distribution is transformed into a tractable prior (e.g., standard Gaussian).

The learned generative trajectory that iteratively denoises a sample of pure noise, reversing the forward SDE to synthesize new data from the target distribution. This is the solution to the reverse-time SDE.

• Training involves learning a score function or a noise prediction network to approximate the drift term of the reverse SDE.
• Sampling simulates the reverse SDE starting from noise, using numerical solvers like Euler-Maruyama.
• The challenge is that the reverse SDE depends on the score of the data distribution, which must be estimated from data.

A training objective where a neural network (score network) learns to estimate the gradient of the log probability density of the data—the score function ∇_x log p(x).

• The score points toward regions of higher data density.
• In the SDE framework, the reverse process drift term is f(x,t) - g(t)² ∇_x log p_t(x). Thus, learning the score is essential for reversing the diffusion.
• Denoising Score Matching is a practical variant that trains the network to predict the noise added during the forward process, which is proportional to the score.

An ordinary differential equation (ODE) derived from the reverse-time SDE of a diffusion model. It describes a deterministic sampling trajectory that shares the same marginal probability distributions as the stochastic process.

• Formulation: Given the reverse SDE dx = [f(x,t) - g(t)² ∇_x log p_t(x)]dt + g(t)dw, the corresponding ODE is dx = [f(x,t) - ½ g(t)² ∇_x log p_t(x)]dt.
• Benefits: Enables fast, deterministic samplers, exact likelihood computation, and latent space interpolation.
• Connection: This ODE is the foundation for Consistency Models, which learn to map any point on an ODE trajectory directly to its origin.

A predefined function that controls the variance of the Gaussian noise added at each step in the forward SDE, determining the progression from data to noise.

• In the SDE dx = f(x, t)dt + g(t)dw, the function g(t) is the noise scale.
• Common schedules include linear, cosine, and sigmoid, affecting training stability and sampling speed.
• A Variance-Exploding schedule lets the variance grow unbounded, while a Variance-Preserving schedule constrains total variance to 1.
• The schedule is a critical design choice impacting the curvature of the probability path and the difficulty of the reverse process.

A class of generative models that learn to map any point on a Probability Flow ODE trajectory directly back to its origin (the data point), enabling one or few-step generation.

• Core Idea: Enforce self-consistency—if you apply the ODE solver from two different points on the same trajectory to the same end time, you should arrive at the same point.
• Training: Can be distilled from a pretrained diffusion model (Consistency Distillation) or trained from scratch (Consistency Training).
• Result: A model that performs high-quality generation in minimal steps by leveraging the deterministic ODE framework derived from the SDE.