Kullback-Leibler Divergence

KL divergence is directional and does not satisfy the symmetry axiom of a true distance metric: D_KL(P || Q) ≠ D_KL(Q || P).

• Forward KL (M-projection): Minimizing D_KL(P || Q) forces Q to cover all modes of P, producing a mean-seeking approximation that averages over ambiguous regions.
• Reverse KL (I-projection): Minimizing D_KL(Q || P) forces Q to fit a single mode of P, producing a mode-seeking approximation that avoids low-probability regions.

In variational inference for grid state estimation, the choice of direction critically impacts whether the approximate posterior under-represents rare fault conditions or over-smooths multimodal load distributions.

Gibbs' inequality guarantees that D_KL(P || Q) ≥ 0 for all distributions P and Q, with equality holding if and only if P = Q almost everywhere.

This property makes KL divergence a valid measure of information gain:

• A value of 0 indicates identical distributions—no information is lost by substituting Q for P.
• Any positive value quantifies the excess surprise, measured in nats (natural log) or bits (log base 2), incurred when observing data generated by P while assuming model Q.

In probabilistic power flow, this lower bound provides a rigorous stopping criterion for iterative distribution-fitting algorithms like expectation propagation.

KL divergence is jointly convex in both arguments: for any two pairs of distributions (P₁, Q₁) and (P₂, Q₂) and any λ ∈ [0,1], the divergence of the mixture is bounded above by the mixture of divergences.

This property ensures:

• Unique global minima in variational optimization problems—gradient descent on the evidence lower bound (ELBO) converges to a single optimum.
• Stable numerical behavior when fitting Gaussian mixture models to non-normal renewable generation forecasts.
• Compatibility with convex stochastic programming formulations used in chance-constrained optimal power flow.

The convex structure guarantees that computational resources spent on KL minimization are not wasted on local minima.

KL divergence is defined only when P is absolutely continuous with respect to Q—meaning Q(x) = 0 implies P(x) = 0 for all x.

If Q assigns zero probability to any region where P has non-zero mass, the divergence becomes infinite:

• This enforces that the approximating distribution Q must have support covering the true distribution P.
• In practice, this prevents degenerate solutions where a fitted Gaussian assigns zero variance to a dimension that actually exhibits uncertainty.

For grid applications using polynomial chaos expansion, this constraint ensures that surrogate models do not artificially eliminate tail risk from wind speed distributions, preserving the integrity of Conditional Value at Risk (CVaR) calculations.

For joint distributions, KL divergence decomposes according to the chain rule of relative entropy:

D_KL(P(X,Y) || Q(X,Y)) = D_KL(P(X) || Q(X)) + E_{x~P}[D_KL(P(Y|X) || Q(Y|X))]

This additive property enables:

• Sequential inference: The total approximation error in a multi-stage grid model (e.g., generation → transmission → distribution) equals the sum of stage-wise divergences.
• Conditional independence testing: If two variables are independent under P but dependent under Q, the divergence isolates the spurious correlation.
• Factorized variational families: Mean-field approximations exploit this decomposition to optimize each factor independently while monitoring the total information loss.

In Bayesian state estimation with Markov structure, the chain rule justifies recursive filtering where KL loss accumulates predictably across time steps.

KL divergence is the difference between cross-entropy and entropy:

D_KL(P || Q) = H(P, Q) - H(P)

Where:

• H(P, Q) = -E_{x~P}[log Q(x)] is the cross-entropy, measuring the average code length needed to encode samples from P using a code optimized for Q.
• H(P) = -E_{x~P}[log P(x)] is the entropy of P, the theoretical minimum average code length.

This decomposition reveals that minimizing KL divergence is equivalent to minimizing cross-entropy—the standard loss function for neural network training. In renewable forecasting, this connects the statistical divergence of predicted vs. actual wind distributions directly to the negative log-likelihood optimization objective used during model fitting.

The most common loss function in classification tasks, directly derived from KL divergence. Minimizing cross-entropy between the true distribution P and the predicted distribution Q is equivalent to minimizing the KL divergence D_KL(P || Q), since the entropy of the true distribution H(P) is a constant offset.

• Formula: H(P, Q) = H(P) + D_KL(P || Q)
• Practical Use: Training neural networks to output class probabilities that match one-hot encoded ground truth labels.

The central objective function in Variational Inference. KL divergence defines the gap between the ELBO and the true marginal log-likelihood. Maximizing the ELBO is mathematically identical to minimizing the KL divergence between the variational approximation q(z) and the true intractable posterior p(z|x).

• Decomposition: log p(x) = ELBO + D_KL(q(z) || p(z|x))
• Role: Drives the optimization in Variational Autoencoders (VAEs) to learn compressed latent representations.

The foundational measure of average uncertainty or information content in a random variable, defined as H(P) = -∑ P(x) log P(x). KL divergence extends this concept to measure the relative uncertainty between two distributions.

• Relationship: KL divergence can be expressed as the difference between cross-entropy and entropy: D_KL(P || Q) = H(P, Q) - H(P).
• Interpretation: The extra number of bits required to encode samples from P using an optimal code designed for Q.

A symmetrized and smoothed version of KL divergence that addresses the asymmetry and undefined regions of the original metric. It calculates the divergence of each distribution from their mixture average M = (P+Q)/2.

• Formula: JSD(P || Q) = 0.5 * D_KL(P || M) + 0.5 * D_KL(Q || M)
• Advantage: Always bounded between 0 and log(2), making it a true distance metric when its square root is taken. Used extensively in Generative Adversarial Networks (GANs).

A general class of divergence measures that includes KL divergence as a specific instance. Any convex function f with f(1)=0 defines an f-divergence: D_f(P || Q) = ∫ Q(x) f(P(x)/Q(x)) dx.

• KL Divergence: Generated by f(t) = t log t.
• Other Members: Total Variation Distance (f(t) = |t-1|/2), Pearson χ² (f(t) = (t-1)²), and Hellinger distance. This framework unifies many distribution comparison techniques.

The standard statistical method for fitting model parameters θ by maximizing the probability of observing the empirical data. This process is formally equivalent to minimizing the KL divergence D_KL(P_data || P_model) between the empirical data distribution and the parametric model.

• Connection: argmin_θ D_KL(P_data || P_model) = argmax_θ E_{x~P_data}[log P_model(x|θ)]
• Implication: Training a model by maximizing log-likelihood is an act of minimizing the forward KL divergence.