Score Function

The score function is formally defined as the gradient of the log probability density function with respect to the data: ∇ₓ log p(x). This vector points in the direction where the log-likelihood of the data increases most rapidly.

• Direction: Points towards regions of higher data density.
• Magnitude: Indicates the steepness of the probability landscape.
• Zero-Mean Property: The expected value of the score over the data distribution is zero: Eₚ₍ₓ₎[∇ₓ log p(x)] = 0.

A critical property is that the score depends only on the gradient of the log-density, not the density itself. This means it is invariant to the normalizing constant (partition function).

• Benefit: Models can be trained on unnormalized probability models, bypassing the often intractable computation of the partition function.
• Application: This is the core advantage exploited in score matching and noise-contrastive estimation training objectives.

In denoising diffusion models, the score function is directly proportional to the optimal noise prediction for a given noisy data point. For a noisy sample xₜ at timestep t, the relationship is:

∇ₓₜ log p(xₜ) = - ε / √(1 - ᾱₜ)

where ε is the true noise added to the original data and ᾱₜ is the cumulative noise schedule product.

• This equivalence allows the score network to be trained to predict noise (ε_θ), a more stable objective than directly regressing the score.

Stein's identity provides a way to compute expectations involving the score without knowing the true data distribution p(x). It states that for a sufficiently smooth test function h(x):

Eₚ₍ₓ₎[∇ₓ log p(x) h(x)ᵀ + ∇ₓ h(x)] = 0

• Implication: This identity underpins the score matching objective proposed by Hyvärinen (2005).
• Training: By minimizing the Fisher divergence between the model and data score, the model's score network learns to approximate ∇ₓ log p(x) using only samples from p(x).

The score function enables sampling via Langevin Dynamics, an iterative Markov Chain Monte Carlo (MCMC) process. Starting from a random sample x₀ ~ π(x), samples are drawn by:

xᵢ₊₁ = xᵢ + (γ/2) ∇ₓ log p(xᵢ) + √γ zᵢ

where γ is a step size and zᵢ ~ N(0, I) is noise.

• Mechanism: The score term ∇ₓ log p(x) drifts the sample toward high-density regions.
• The noise term √γ zᵢ provides stochasticity for exploration and ensures the chain converges to the true distribution p(x).

Real-world data often lies on a low-dimensional manifold within the high-dimensional ambient space. This creates a fundamental challenge for the score function.

• Problem: The true data distribution p(x) is singular (zero probability) off the manifold. The log-density is undefined, and the score is unbounded or ill-behaved in these vast, empty regions.
• Solution: Score-based models circumvent this by always working with noise-perturbed data distributions p_σ(x) = ∫ p(y) N(x; y, σ²I) dy. Adding noise with variance σ² smoothes the distribution, giving it full support in ℝᴺ and making the score well-defined everywhere.

The primary training objective used to estimate the score function without needing to compute the intractable data distribution's normalizing constant. A neural network, called a score network, is trained to match the true gradient of the log probability density. Key variants include:

• Denoising Score Matching: Trains the network to estimate the score of a noise-perturbed data distribution, which is more tractable.
• Sliced Score Matching: Projects the score onto random vectors to reduce computational cost. This objective is the theoretical backbone for training models like Denoising Diffusion Probabilistic Models (DDPMs).

A Markov Chain Monte Carlo (MCMC) sampling algorithm that uses the score function to draw samples from a probability distribution. Starting from a random point, it iteratively takes steps in the direction of the score (toward higher data density) while injecting a controlled amount of Gaussian noise. The update rule is: x_{t+1} = x_t + ε * score(x_t) + √(2ε) * z_t, where ε is a step size and z_t is noise. This process is the sampling mechanism in score-based generative models, where the learned score network guides the dynamics.

A common and practical parameterization for the model in diffusion models that is functionally equivalent to a score network. Instead of directly predicting the score ∇_x log p(x), the network is trained to predict the noise component ε that was added to a clean data point x_0 to produce a noisy version x_t. The relationship is given by: score ≈ - (predicted_noise) / (noise_scale). This re-framing often leads to more stable training and is the standard approach in models like DDPM and Stable Diffusion.

A continuous-time framework that generalizes the diffusion process. The forward noising process is described by a diffusion SDE. Crucially, the reverse-time SDE for generating data is defined by the score function. This formulation reveals that sampling (the reverse process) involves solving an SDE where the drift term depends on the score. This perspective connects diffusion models to broader literature on continuous-time generative modeling and enables the use of advanced numerical solvers for sampling.

An ordinary differential equation (ODE) derived from the reverse-time Stochastic Differential Equation (SDE). This ODE describes a deterministic trajectory that has the same marginal probability distributions as the stochastic diffusion process. The drift of this ODE is also defined by the score function. Sampling via this ODE allows for:

• Deterministic generation from a fixed noise input.
• Efficient sampling with fewer steps using adaptive ODE solvers.
• Exact likelihood computation in some cases. It is the basis for DDIM sampling and Consistency Models.

A measurement of the amount of information a random variable carries about an unknown parameter. In the context of the score function, the Fisher information matrix is defined as the expected covariance of the score: I(θ) = E[∇ log p(x;θ) ∇ log p(x;θ)^T]. It quantifies the sensitivity of the probability distribution to changes in parameters. The score function's relationship to Fisher information is fundamental to statistical estimation theory and influences the design of efficient estimators and the analysis of model training dynamics.