Adversarial Loss

The foundational adversarial loss, formulated as a two-player minimax game. The discriminator (D) maximizes the probability of correctly classifying real and fake data, while the generator (G) minimizes the probability that D will be correct.

• Objective: (\min_G \max_D V(D, G) = \mathbb{E}{x\sim p{data}}[\log D(x)] + \mathbb{E}_{z\sim p_z}[\log(1 - D(G(z)))])
• Challenge: The generator's gradient can vanish early in training when the discriminator becomes too confident, a problem known as saturation.

A practical modification to the original minimax loss designed to prevent gradient saturation for the generator. Instead of minimizing (\log(1 - D(G(z)))), the generator maximizes (\log D(G(z))).

• Generator Objective: (\max_G \mathbb{E}_{z\sim p_z}[\log D(G(z))])
• Effect: This provides stronger gradients for the generator early in training, leading to more stable updates. It is the de facto standard loss for the generator in many modern GAN implementations, while the discriminator continues to use the original minimax objective.

Replaces the Jensen-Shannon divergence with the Wasserstein-1 distance (Earth Mover's Distance). It uses a critic network (instead of a discriminator) that outputs a scalar score rather than a probability.

• Critic Objective: (\max_{D \in \mathcal{D}} \mathbb{E}{x\sim p{data}}[D(x)] - \mathbb{E}_{z\sim p_z}[D(G(z))])
• Key Innovation: The critic must be a 1-Lipschitz function, enforced via weight clipping or gradient penalty (WGAN-GP). This loss correlates with sample quality, providing a meaningful training signal and greatly improving training stability.

Substitutes the binary cross-entropy loss with a least squares (L2) loss. This penalizes samples based on their distance from the decision boundary, not just their classification.

• Generator Objective: (\min_G \frac{1}{2} \mathbb{E}_{z\sim p_z}[(D(G(z)) - 1)^2])
• Discriminator Objective: (\min_D \frac{1}{2} \mathbb{E}{x\sim p{data}}[(D(x) - 1)^2] + \frac{1}{2} \mathbb{E}_{z\sim p_z}[(D(G(z)))^2])
• Benefit: It pulls generated samples toward the manifold of real data, often generating higher quality results and reducing the problem of vanishing gradients.

A loss function adapted from support vector machines, commonly used in GANs like BigGAN and SAGAN. It provides a robust margin for the discriminator's decisions.

• Discriminator Loss: (\mathbb{E}{x\sim p{data}}[\max(0, 1 - D(x))] + \mathbb{E}_{z\sim p_z}[\max(0, 1 + D(G(z)))])
• Generator Loss: (-\mathbb{E}_{z\sim p_z}[D(G(z))])
• Characteristic: The hinge loss encourages a clear separation margin, which can lead to more stable training dynamics and is less sensitive to outliers compared to the original loss.

A supervisory loss used in CycleGAN for unpaired image-to-image translation. It enforces that translating a sample from domain A to B and back to A should reconstruct the original sample.

• Formulation: (\mathcal{L}{cyc}(G, F) = \mathbb{E}{x\sim p_{data}(x)}[|F(G(x)) - x|1] + \mathbb{E}{y\sim p_{data}(y)}[|G(F(y)) - y|_1])
• Purpose: This reconstruction constraint prevents the two adversarial generators (G and F) from producing arbitrary mappings, ensuring the learned translation is semantically meaningful and preserves the input's core content.

The minimax game is the foundational mathematical framework for GAN training, formalized as a two-player zero-sum game. The generator aims to minimize the probability of the discriminator being correct, while the discriminator aims to maximize it. This is expressed in the original GAN objective function: min_G max_D V(D, G) = E_{x~p_data}[log D(x)] + E_{z~p_z}[log(1 - D(G(z)))]. The adversarial loss directly implements this game, seeking a Nash Equilibrium where neither player can improve their outcome unilaterally.

The non-saturating loss is a practical modification to the original minimax loss designed to solve the gradient vanishing problem. In the original formulation, the generator's loss (log(1 - D(G(z)))) can saturate early in training when the discriminator is too effective, providing near-zero gradients. The non-saturating variant flips the generator's objective to maximize log(D(G(z))) instead. This provides stronger gradients when the generator is performing poorly, leading to more stable training and is the de facto standard in many modern GAN implementations.

Wasserstein loss, used in Wasserstein GANs (WGAN), replaces the Jensen-Shannon divergence with the Earth Mover's Distance (Wasserstein-1 distance). Instead of a discriminator outputting a probability, a critic network outputs a scalar score. The loss is calculated as the difference between the critic's scores for real and fake data: L = E[D(x_real)] - E[D(G(z))]. Key advantages include:

• Meaningful loss metric: The loss value correlates with generation quality.
• Improved stability: Mitigates mode collapse and vanishing gradients.
• Requires weight clipping or spectral normalization to enforce a Lipschitz constraint on the critic.

Hinge loss is an alternative adversarial objective used in GAN variants like BigGAN and StyleGAN2. It provides a more robust optimization landscape than the standard logistic loss. For the discriminator/critic, it encourages a margin between real and fake samples: L_D = E[max(0, 1 - D(x_real))] + E[max(0, 1 + D(G(z)))]. For the generator, it simply aims to maximize the critic's output for fake data: L_G = -E[D(G(z))]. This formulation often leads to faster convergence and higher training stability, as it is less sensitive to outliers and provides a linear penalty beyond the margin.

Cycle-consistency loss is a reconstruction loss used in CycleGAN for unpaired image-to-image translation. It is not a direct adversarial loss but works in tandem with it to enforce semantic preservation. The loss ensures that translating a sample from domain A to B and back again (A → B → A') reconstructs the original sample. It is calculated as the L1 distance between the original and reconstructed image: L_cyc = E[||x - G_BA(G_AB(x))||_1]. This auxiliary objective provides the necessary constraint for the adversarial networks to learn meaningful, bidirectional mappings without paired training data.

Feature matching loss is a regularization technique introduced to improve GAN training stability. Instead of directly using the discriminator's final output to train the generator, this loss matches the intermediate feature representations (activations) of real and fake data within the discriminator. The generator's objective becomes to minimize the L2 distance between the expected features: L_G = ||E[f(x_real)] - E[f(G(z))]||_2^2. This encourages the generator to produce data that matches the statistics of real data at a feature level, which can prevent mode collapse and lead to more diverse outputs, especially in early training phases.