Noise Schedule

The core of a noise schedule is the variance schedule, a sequence of values (β₁, ..., β_T) that determines the amount of Gaussian noise added at each timestep t. These values are typically predefined and satisfy 0 < β_t < 1. Common schedules include:

• Linear Schedule: β_t increases linearly from a small value (e.g., 0.0001) to a larger value (e.g., 0.02).
• Cosine Schedule: β_t follows a cosine function, leading to a more gradual noise addition, which often improves sample quality.
• Sigmoid Schedule: β_t follows a sigmoid curve, adding noise slowly at the start and end of the process, and faster in the middle. The choice of schedule controls how quickly data is corrupted into pure noise.

In practice, diffusion models operate using the cumulative product of noise retention, defined as ᾱ_t = Π_{s=1}^{t} (1 - β_s). This value represents the total proportion of the original data signal remaining after t steps of noise addition. A noisy sample x_t is directly computed from the original data x_0 and noise ε via the reparameterization trick: x_t = sqrt(ᾱ_t) * x_0 + sqrt(1 - ᾱ_t) * ε. The schedule of ᾱ_t monotonically decreases from ~1 to ~0, defining the signal-to-noise ratio over time. This formulation is crucial for efficient training, as it allows sampling x_t for any t in a single step without iterating through the Markov chain.

The Signal-to-Noise Ratio (SNR) is a derived characteristic, defined as SNR(t) = ᾱ_t / (1 - ᾱ_t). It quantifies the relative strength of the data signal versus the added noise at timestep t. The noise schedule dictates the decay of SNR over time.

• A rapidly decaying SNR (e.g., linear schedule) means the data becomes predominantly noisy early in the forward process.
• A slowly decaying SNR (e.g., cosine schedule) preserves more signal for longer, which can simplify the denoising task for the model during the reverse process. The SNR trajectory is a fundamental design consideration, influencing the difficulty of the score-matching objective learned by the model.

Noise schedules are categorized by their effect on the total variance of the noisy data distribution.

• Variance-Preserving (VP) Schedules: Constrain the total variance of x_t to remain constant (typically 1) across all timesteps. This is achieved by ensuring β_t is small. The DDPM model uses a VP schedule. The forward process maps data to a standard normal distribution N(0, I).
• Variance-Exploding (VE) Schedules: Allow the variance of x_t to grow significantly over time, often following a geometric progression. The data is corrupted towards a distribution with very high variance. Score-based generative modeling (SGM) often uses VE schedules. The VP formulation is more common in modern image generation models due to its numerical stability.

In the continuous-time limit, the noise schedule is described by continuous functions β(t) and ᾱ(t), where t ranges from 0 to 1 (or 0 to T). This framework, formalized via Stochastic Differential Equations (SDEs), unifies discrete schedules. The drift and diffusion coefficients of the SDE are defined by the schedule. This allows for the use of advanced numerical solvers during sampling and enables the derivation of a deterministic Probability Flow ODE that generates samples without stochastic noise. The continuous view is essential for understanding and developing fast samplers like DDIM and for training consistency models.

The noise schedule is not just for training; it dictates the sampling process. Fast samplers (e.g., DDIM, DPM-Solver) work by strategically evaluating the model's noise prediction only at a subset of the original timesteps, following a discretization of the continuous schedule. The chosen sub-sequence must respect the schedule's curvature for good results. Furthermore, for model distillation techniques that aim to reduce sampling steps (e.g., Progressive Distillation, Consistency Models), the student model is trained to match outputs on a new, sparser timestep sequence derived from the teacher's original schedule. An ill-chosen schedule can make distillation unstable or degrade final sample quality.

The fixed Markov chain that applies the noise schedule to gradually corrupt a data sample into pure Gaussian noise over T timesteps. It is defined by a variance schedule (beta_t), where each step adds noise scaled by the schedule's parameters.

• Mathematical Definition: q(x_t | x_{t-1}) = N(x_t; sqrt(1 - beta_t) * x_{t-1}, beta_t * I)
• Role of Schedule: The sequence of beta_t values, often increasing over time, dictates the rate of information destruction.
• Cumulative Effect: The total noise added by timestep t is a function of all previous schedule parameters, often denoted as alpha_bar_t.

A specific parameterization of the forward diffusion process where the noise schedule is designed so that the total variance of the noisy sample remains constant (typically 1) across all timesteps. This constraint simplifies the model's behavior and is used in foundational works like DDPM.

• Key Property: Ensures x_T has a unit Gaussian distribution.
• Schedule Relation: The beta_t values in the schedule must satisfy the constraint: alpha_bar_t + (1 - alpha_bar_t) = 1, where alpha_t = 1 - beta_t.
• Contrast: Differs from a variance-exploding process, where variance increases without bound.

The neural network (e.g., a U-Net) trained to predict the noise component epsilon added to a sample at a given timestep t. Its training objective is directly conditioned on the noise schedule parameters.

• Training Objective: L = E[|| epsilon - epsilon_theta(x_t, t) ||^2], where x_t is computed via the forward process using the schedule.
• Schedule Dependence: The network receives the timestep t (or its embedding) as input, allowing it to learn the noise distribution specific to each schedule-defined corruption level.
• Equivalence: Predicting noise is functionally equivalent to estimating the score function under a specific parameterization.

A continuous-time framework that generalizes the discrete noise schedule into a differential equation with a stochastic (noise) term. The forward SDE describes a continuous path from data to noise.

• Forward SDE: dx = f(x, t)dt + g(t)dw, where w is a Wiener process.
• Schedule as g(t): The diffusion coefficient g(t) defines the continuous-time analog of the discrete noise schedule's beta_t.
• Flexibility: The SDE view allows for the design of different schedule families (e.g., Variance Preserving, Variance Exploding) and enables advanced sampling methods via its corresponding reverse SDE.

An ordinary differential equation (ODE) derived from the reverse-time SDE. It describes a deterministic trajectory that shares the same marginal distributions as the stochastic diffusion process defined by the noise schedule.

• Deterministic Sampling: Enables generation via black-box ODE solvers (e.g., Euler, Runge-Kutta).
• Schedule Dependence: The drift term of this ODE is a function of the learned score, which itself was trained under a specific noise schedule.
• Benefit: Allows for faster sampling with adaptive step sizes and enables exact likelihood computation and latent code manipulation.

The standard stochastic sampling algorithm for diffusion models, where each reverse step denoises the sample and then adds back a small amount of new noise. The amount of noise added at each step is determined by the noise schedule parameters (beta_t or sigma_t).

• Procedure: x_{t-1} ~ p_theta(x_{t-1} | x_t). This distribution has a mean from the denoising network and a variance defined by the schedule.
• Schedule's Role: The reverse process variance schedule is typically set as a function of the forward beta_t (e.g., in DDPM, it is set to tilde_beta_t).
• Contrast: Differs from deterministic samplers like DDIM, which use the schedule but set this additive noise to zero.