Wasserstein Distance

The Wasserstein distance is intuitively understood as the minimum cost required to transform one probability distribution into another. Imagine two piles of dirt: the distance is the amount of dirt multiplied by the average distance it must be moved. This geometric interpretation directly contrasts with f-divergences like KL divergence, which only compare pointwise likelihood ratios and can explode when distributions have non-overlapping support.

Unlike Jensen-Shannon divergence, which can saturate and produce vanishing gradients when the generator and real data distributions are far apart, the Wasserstein distance provides smooth, meaningful gradients everywhere. This property is critical for training Generative Adversarial Networks (GANs), as it eliminates the mode collapse and training instability caused by a discriminator that becomes perfect too quickly, offering the generator a clear path to improvement.

Computing the exact Wasserstein distance is computationally intractable. The Kantorovich-Rubinstein duality provides a tractable alternative, reformulating the problem as:

• Finding a 1-Lipschitz function that maximizes the difference in expectations under the two distributions.
• This is the theoretical foundation for the Wasserstein GAN (WGAN), where the discriminator (critic) learns this function.
• The Lipschitz constraint is typically enforced via weight clipping or a gradient penalty.

Wasserstein distance is sensitive to the underlying geometry of the data manifold. It naturally captures how far apart two distributions are in the metric space, not just how much their probabilities differ. For example, the distance between a delta function at 0 and a delta function at θ is exactly |θ|. This makes it ideal for tasks like domain adaptation and representation learning, where preserving the spatial structure of the data is essential for meaningful comparisons.

Beyond a single scalar distance, the Wasserstein metric provides an explicit optimal transport plan—a coupling matrix that shows exactly how probability mass from the source distribution should be mapped to the target distribution. This plan is a powerful analytical tool in its own right, used for:

• Color transfer between images
• Single-cell genomics alignment
• Document similarity via Word Mover's Distance

The Wasserstein distance is a true metric in the mathematical sense, satisfying all required axioms:

• Positivity: W(P, Q) ≥ 0, and W(P, Q) = 0 if and only if P = Q.
• Symmetry: W(P, Q) = W(Q, P).
• Triangle Inequality: W(P, R) ≤ W(P, Q) + W(Q, R). This formal structure allows it to be used in convergence proofs and as a loss function in variational inference, where the geometry of the metric space is preserved.

A deep learning architecture where two neural networks—a generator and a discriminator—compete adversarially. The Wasserstein GAN (WGAN) variant replaces the standard discriminator with a critic that estimates the Earth Mover's distance, dramatically improving training stability by providing meaningful gradients even when the real and generated distributions have non-overlapping support.

• Standard GANs minimize Jensen-Shannon divergence, which can saturate
• WGAN uses weight clipping or gradient penalty to enforce the 1-Lipschitz constraint
• Results in smoother convergence and reduced mode collapse

A catastrophic failure condition in GAN training where the generator learns to produce only a narrow subset of the data distribution, ignoring entire modes of variation. Wasserstein distance provides a theoretically grounded remedy because it remains continuous and differentiable even when distributions have disjoint supports, unlike KL or JS divergence.

• Generator maps multiple latent codes to identical outputs
• WGAN's critic provides stable gradients across the full distribution
• Enables the generator to explore and cover all data modes

The mathematical foundation of Wasserstein distance, originating from Gaspard Monge's 1781 problem of moving earth with minimal cost. Optimal transport finds the most efficient plan to transform one probability distribution into another, where the cost is defined by a ground metric (typically Euclidean distance).

• Kantorovich relaxation makes the problem tractable via linear programming
• The Wasserstein-p distance generalizes to different cost exponents
• Sinkhorn distances add entropic regularization for computational efficiency

An entropically regularized approximation of Wasserstein distance that adds an entropy penalty to the optimal transport objective. This regularization makes the problem strictly convex and solvable via the Sinkhorn-Knopp algorithm, reducing computational complexity from O(n³ log n) to approximately O(n²).

• Controlled by a regularization parameter λ (lower = closer to true Wasserstein)
• Enables GPU-accelerated batch computation via iterative matrix scaling
• Widely used in domain adaptation and single-cell genomics

The dual formulation of the Wasserstein-1 distance that enables its practical computation in machine learning. It expresses the distance as a supremum over 1-Lipschitz functions, which neural networks can approximate as critics. This duality is the theoretical backbone of WGAN.

• Primal: minimize transport cost over coupling matrices
• Dual: maximize expected critic difference over Lipschitz functions
• Enforces the 1-Lipschitz constraint via weight clipping or gradient penalty

The degree to which synthetic data preserves the statistical properties of real data, including univariate distributions, multivariate correlations, and boundary adherence. Wasserstein distance serves as a rigorous metric for quantifying fidelity by measuring the minimal geometric distance between real and synthetic distributions in feature space.

• Lower Wasserstein distance indicates higher fidelity
• Captures both marginal and joint distribution differences
• Complements privacy metrics in synthetic data quality reports