Optimal Transport for Domain Adaptation

The Wasserstein distance (Earth Mover's Distance) is the central metric in Optimal Transport for DA. Unlike KL divergence, it provides a meaningful distance between distributions even when they have non-overlapping support. It is defined as the minimum cost of transporting mass from the source distribution to match the target distribution. This makes it particularly suitable for measuring domain shift and constructing alignment losses that are smoother and provide more useful gradients for training.

The transport cost defines the geometry of the alignment. It is typically a distance metric (e.g., Euclidean, cosine) between individual data points or their features.

• A common choice is the squared Euclidean distance in a feature space learned by a neural network.
• The design of this cost function directly controls what properties are preserved during adaptation. For example, using a semantic feature distance encourages aligning samples from the same class across domains.
• The overall OT problem seeks a coupling (transport plan) that minimizes the total cost of moving the source distribution to the target.

The original OT problem is a linear program that scales poorly. Entropic regularization is almost always applied, adding a convex penalty term based on the KL divergence of the transport plan. This results in the Sinkhorn algorithm, which provides a fast, differentiable approximation of the Wasserstein distance.

• Key hyperparameter: Regularization strength (ε). A higher ε speeds up computation but blurs the coupling.
• This differentiability is crucial, as it allows the OT loss to be integrated into end-to-end deep learning pipelines for unsupervised domain adaptation (UDA).

OT can align distributions in both feature space and label space.

• Feature Space OT: Aligns the deep feature representations extracted by a neural network, encouraging domain-invariant features.
• Label Space OT: Uses classifier predictions (e.g., softmax outputs) as distributions. Minimizing the Wasserstein distance between prediction distributions on source and target data encourages the model to make similar predictions on both domains, effectively performing adaptation.
• Advanced methods like JDOT (Joint Distribution Optimal Transport) perform alignment in the product space of features and labels.

A naive application of OT treats all source and target samples equally, which can fail if the class prior distributions differ between domains (a common scenario).

• Class-conditional OT addresses this by computing transport plans separately per class or by re-weighting the marginal distributions of the OT problem based on estimated target class proportions.
• This ensures that mass is transported between samples of the same semantic class, preventing negative transfer where a source class incorrectly aligns with a different target class.

There is a deep theoretical link between OT and adversarial domain adaptation methods like DANN. The Kantorovich-Rubinstein duality shows that the Wasserstein distance can be expressed as a supremum over critic functions (1-Lipschitz).

• In practice, this leads to Wasserstein GAN-inspired approaches for DA, where a critic (discriminator) is trained to estimate the Wasserstein distance, and the feature extractor is trained to minimize it.
• This connection provides a robust, gradient-based alternative to min-max adversarial training with standard classifiers.

Domain shift refers to the change in the underlying joint probability distribution P(X, Y) between a model's training environment (source domain) and its deployment environment (target domain). This statistical discrepancy is the fundamental problem domain adaptation aims to solve.

• Causes: Changes in data collection (sensor, camera), environment (lighting, weather), or population (demographics, user behavior).
• Impact: Leads to degraded model performance (e.g., accuracy drop) on the target data despite good source performance.
• Types: Includes covariate shift (P(X) changes, P(Y|X) stable), label shift (P(Y) changes, P(X|Y) stable), and concept shift (P(Y|X) changes).

The Wasserstein distance (Earth Mover's Distance) is a metric from optimal transport theory that measures the minimum cost of transforming one probability distribution into another. It is the core objective minimized in optimal transport-based domain adaptation.

• Formally: For distributions μ and ν, it's the infimum cost over all joint distributions (transport plans) with marginals μ and ν.
• Properties: Provides a meaningful gradient even when distributions have disjoint support (unlike KL divergence), making it suitable for training.
• Use in DA: The Wasserstein-1 distance is often used in losses (e.g., Wasserstein GAN) to align source and target feature distributions, encouraging domain-invariant representations.

A Domain-Adversarial Neural Network is an adversarial learning architecture for learning domain-invariant features. It uses a gradient reversal layer (GRL) to train a feature extractor to produce representations that confuse a concurrent domain classifier.

• Mechanism: A three-component network (feature extractor, label predictor, domain classifier) is trained with two conflicting objectives: accurate label prediction and indistinguishable domain features.
• Contrast with OT: While DANN uses adversarial discrimination, optimal transport methods use explicit distribution distance minimization. DANN can be seen as minimizing a Jensen-Shannon divergence between domains.

Maximum Mean Discrepancy is a kernel-based statistical test used to measure the distance between two probability distributions. It is a common alternative to Wasserstein distance for distribution alignment in domain adaptation.

• Calculation: MMD is defined as the distance between the mean embeddings of the distributions in a reproducing kernel Hilbert space (RKHS).
• Application: Used as a regularization loss to minimize the discrepancy between source and target feature distributions extracted by a neural network.
• Comparison to OT: MMD is generally computationally cheaper than solving a full optimal transport problem but may be less powerful in capturing geometric relationships between distributions.

Fréchet Inception Distance is a metric for evaluating the quality of generated images, but its principle is deeply connected to optimal transport. It calculates the Wasserstein-2 distance between multivariate Gaussian distributions fitted to the feature activations of real and synthetic images.

• How it works: Images are passed through a pre-trained Inception-v3 network. The mean and covariance of the resulting feature embeddings are calculated for both sets. The FID is the Fréchet distance between these two Gaussians.
• Relevance to OT: FID is a practical, efficient approximation of the Wasserstein-2 distance for high-dimensional data, showcasing OT's utility in evaluating distribution alignment—the core goal of domain adaptation.

Sim-to-real transfer is a critical application domain for optimal transport-based adaptation. It involves training a model in a simulated (synthetic) source domain and adapting it for deployment in the physical (real) target domain.

• The Challenge: The reality gap—discrepancies in visuals, physics, and sensor noise between simulation and reality.
• OT's Role: Optimal transport can find a mapping between synthetic and real data distributions, effectively "translating" simulated features or states to their real-world counterparts or aligning their statistical properties.
• Related Technique: Domain randomization, which varies simulation parameters widely, is a complementary technique that creates a broad, diverse source distribution, making the subsequent alignment via OT more robust.