Wasserstein Distance

Formally, the p-th Wasserstein distance between distributions P and Q is the minimum cost of transporting probability mass from P to Q, where the cost of moving a unit of mass from x to y is given by the distance metric d(x, y)^p. Unlike f-divergences such as KL divergence, it leverages the underlying geometry of the sample space. This means it provides a meaningful distance even when the supports of the distributions do not overlap, a critical failure point for other metrics.

The 1-Wasserstein distance (W1) can be computed using its dual formulation, which is central to the Wasserstein GAN (WGAN). The duality states that W1(P, Q) equals the supremum over all 1-Lipschitz functions f of the difference in expectations under P and Q. This transforms a complex transport problem into a maximization problem over a critic function, enabling gradient-based optimization in deep learning.

The Wasserstein distance is a true metric, satisfying key mathematical axioms:

• Positive Definiteness: W(P, Q) >= 0, and equals 0 if and only if P = Q.
• Symmetry: W(P, Q) = W(Q, P), unlike the asymmetric KL divergence.
• Triangle Inequality: W(P, R) <= W(P, Q) + W(Q, R). These properties ensure a well-behaved, geometrically consistent loss landscape for training models.

A sequence of probability distributions converges in the Wasserstein metric if and only if it converges weakly and the sequence has uniformly integrable p-th moments. This property makes Wasserstein distance sensitive to the geometry of the distribution, not just pointwise probabilities. For example, the distance between a Dirac delta at 0 and a Dirac delta at θ smoothly increases with θ, unlike the sudden jump to infinity seen with KL divergence.

In channel-robust feature learning, Wasserstein distance is used to align the feature distributions of the source and target domains. By minimizing the Wasserstein distance between the latent representations of signals from different channel conditions, the model learns to extract channel-invariant features. This is often implemented via a critic network that estimates the dual potential, pulling domain distributions together in the embedding space without requiring paired data.

Computing the exact Wasserstein distance suffers from high computational cost (O(n^3 log n)) and biased sample gradients. The Sinkhorn divergence addresses this by adding an entropic regularization term to the optimal transport problem. This makes the problem strictly convex and solvable via the highly efficient Sinkhorn-Knopp algorithm, enabling differentiable, scalable computation of a Wasserstein-like metric for large-scale deep learning applications.

A kernel-based statistical measure of the distance between two probability distributions. Unlike Wasserstein distance, which solves an optimal transport problem, MMD compares distributions by computing the squared difference between their mean embeddings in a reproducing kernel Hilbert space (RKHS). Key differences:

• MMD is computationally simpler but less sensitive to geometric structure
• Wasserstein captures the underlying metric space geometry
• Both are used as domain adaptation regularization terms to align source and target feature distributions

A technique that trains neural networks to learn features that are discriminative for the primary task while being indistinguishable across different domains. The Gradient Reversal Layer (GRL) is a key architectural component that reverses gradients during backpropagation, maximizing domain classifier loss. Relationship to Wasserstein:

• Adversarial methods implicitly minimize a Jensen-Shannon divergence
• Wasserstein GANs use the Wasserstein distance for more stable adversarial training
• Both approaches force the feature extractor to produce domain-invariant representations

Correlation Alignment (CORAL) loss aligns the second-order statistics of source and target feature distributions by minimizing the Frobenius norm of the difference between their covariance matrices. Comparison with Wasserstein:

• CORAL matches only covariance (second moment)
• Wasserstein matches the full distribution shape
• CORAL is computationally efficient: O(d²) where d is feature dimension
• Wasserstein provides a more complete distribution alignment but at higher computational cost

A self-supervised learning paradigm that trains models to pull representations of similar data points together and push dissimilar ones apart in the embedding space. Connection to optimal transport:

• Contrastive objectives can be viewed as approximating an optimal transport problem
• Sinkhorn distances (entropy-regularized Wasserstein) bridge contrastive and transport-based views
• Both approaches learn channel-invariant features by enforcing structure in the latent space
• Contrastive methods typically require careful negative sampling strategies

The process of separating a learned representation into independent, interpretable factors of variation. In RF fingerprinting, this means isolating device-specific features from channel-induced distortions. Wasserstein's role:

• Wasserstein distance can measure the independence between disentangled factors
• Used to enforce that channel and device representations occupy distinct subspaces
• Enables robust device identification even when channel conditions vary dramatically
• Often combined with variational autoencoders or InfoGAN architectures

An entropy-regularized approximation of the Wasserstein distance that makes optimal transport computationally tractable for high-dimensional problems. Key properties:

• Adds an entropic penalty term to the Kantorovich formulation
• Enables fast computation via the Sinkhorn-Knopp algorithm
• Differentiable and suitable for gradient-based optimization
• Smooths the transport plan, preventing degenerate solutions
• Widely used in domain adaptation when exact Wasserstein is too expensive