Training Stability

Mode collapse is a catastrophic failure mode where the generator learns to produce a limited variety of outputs, often collapsing to generate only one or a few plausible samples. This occurs when the generator exploits a specific weakness in the discriminator, finding a single output that reliably fools it. The generator then ceases to explore the full data distribution.

• Example: A GAN trained on a dataset of animal images might only generate images of dogs, ignoring all other classes.
• Impact: The model fails to capture the diversity of the training data, rendering the generated dataset useless for most applications.
• Mitigations: Techniques include minibatch discrimination, unrolled GANs, and using alternative loss functions like the Wasserstein loss.

Vanishing gradients occur when the discriminator becomes too proficient, providing near-zero gradients to the generator. In the original GAN formulation, if the discriminator perfectly distinguishes real from fake data (outputs 0 for fake, 1 for real), the gradient for the generator's loss function vanishes, halting its learning. This is a form of training saturation.

• Mechanism: The generator's loss (e.g., log(1 - D(G(z)))) flattens as D(G(z)) approaches 0, providing no useful signal for updates.
• Solution: The non-saturating loss flips the generator's objective to maximize log(D(G(z))) instead, providing stronger gradients when the discriminator is confident.

Instead of converging to a Nash Equilibrium, the generator and discriminator often enter a persistent, non-convergent oscillatory state. The networks continuously chase each other's updates without reaching a stable point, causing loss values to oscillate wildly rather than settle.

• Cause: The simultaneous gradient-based optimization of two competing networks in a non-cooperative game is inherently unstable. The loss landscape is non-stationary for each network.
• Indicator: Erratic, non-monotonic loss curves for both networks during training.
• Approaches: Using two-timescale update rule (TTUR), where the generator and discriminator are trained with different learning rates, can help dampen oscillations.

Discriminator overfitting happens when the discriminator memorizes the training dataset rather than learning a general boundary between real and generated distributions. A severely overfit discriminator provides noisy, ungeneralizable feedback to the generator, degrading training.

• Symptoms: The discriminator loss drops to near zero very quickly, while the generator loss remains high and unstable.
• Consequence: The generator receives a meaningless training signal, as it is trying to fool a model that has simply memorized specific examples.
• Prevention: Applying strong regularization techniques to the discriminator, such as dropout, weight decay, label smoothing, and spectral normalization.

GAN training is notoriously sensitive to hyperparameter choices, requiring meticulous tuning for stability. Small changes can lead to complete training failure.

• Critical Parameters: Learning rates for the generator and discriminator, batch size, optimizer choice (Adam is common), and network architecture details.
• Challenge: The optimal settings are highly dataset-dependent and lack robust theoretical guidance, often requiring extensive grid searches.
• Stabilizing Factors: Architectural guidelines from DCGAN (e.g., using strided convolutions, BatchNorm in the generator) and using Wasserstein GAN with Gradient Penalty (WGAN-GP) have reduced but not eliminated this sensitivity.

The loss values reported during GAN training are often poor indicators of actual sample quality and model convergence. A low generator loss does not necessarily correspond to high-quality, diverse outputs.

• Problem: The adversarial loss measures the generator's success in fooling the current discriminator, not the distance to the true data distribution. The discriminator is a moving target.
• Result: Engineers cannot rely on training loss curves to select checkpoints or diagnose issues. Frechet Inception Distance (FID) and Inception Score (IS) are required for external evaluation.
• Implication: Training becomes more experimental and resource-intensive, as models must be evaluated qualitatively and with external metrics throughout the process.

Mode collapse is a catastrophic training failure where the generator learns to produce a very limited variety of outputs, effectively 'collapsing' to generate only one or a few modes of the true data distribution. This occurs when the generator finds a single output that reliably fools the discriminator, causing it to exploit this weakness and stop exploring.

• Example: A GAN trained on a dataset of animal images might only generate pictures of cats, ignoring all other species.
• Impact: It destroys the diversity of the generated dataset, rendering it useless for most applications.

In GAN theory, a Nash Equilibrium is the ideal, stable state of the two-player minimax game. It is reached when the generator produces samples perfectly matching the real data distribution, and the discriminator is maximally uncertain, assigning a probability of exactly 0.5 (a random guess) to every input, real or fake.

• Challenge: The continuous, high-dimensional optimization of neural networks makes converging to this precise equilibrium extremely difficult in practice.
• Outcome: The training dynamics often oscillate around this point rather than settling into it.

Spectral normalization is a weight normalization technique applied to the discriminator (or critic) to enforce a Lipschitz constraint. It controls the maximum gradient magnitude of the discriminator, preventing it from becoming too powerful too quickly, which is a primary cause of unstable gradients and mode collapse.

• Mechanism: It constrains the spectral norm (the largest singular value) of each weight matrix in the network.
• Benefit: This leads to smoother, more predictable gradient signals for the generator, significantly improving training stability. It is a key component in models like Spectral Normalization GAN (SN-GAN).

The Wasserstein GAN (WGAN) is a seminal architecture that reformulates the adversarial objective using the Wasserstein-1 distance (Earth Mover's Distance). Unlike the original GAN loss, this distance provides a continuous and meaningful measure of progress even when the distributions are disjoint.

• Key Changes:
  • Replaces the discriminator with a critic that outputs a scalar score instead of a probability.
  • Uses a linear activation in the critic's output layer.
  • Employs weight clipping or spectral normalization to enforce the Lipschitz constraint.
• Advantage: It largely mitigates mode collapse and vanishing gradients, providing stable training with a loss value that correlates with sample quality.

The non-saturating loss is a practical modification to the original GAN minimax objective designed to prevent gradient saturation. In the original formulation, the generator's loss can saturate (produce near-zero gradients) if the discriminator rejects its samples too confidently early in training.

• Original Loss: Generator minimizes log(1 - D(G(z))). This can saturate.
• Non-Saturating Loss: Generator maximizes log(D(G(z))) instead. This reformulation provides stronger gradients when the generator is performing poorly, encouraging faster and more stable learning.
• Usage: It is the default generator loss in many standard GAN implementations.

Frechet Inception Distance (FID) is the gold-standard metric for quantitatively evaluating GAN stability and output quality. It measures the statistical similarity between the distributions of real and generated images.

• Process:
  1. Embed both real and generated images using a pre-trained Inception-v3 network (up to the pool3 layer).
  2. Model each set of embeddings as a multivariate Gaussian.
  3. Calculate the Frechet distance between the two Gaussians.
• Interpretation: A lower FID score indicates that the generated distribution is closer to the real distribution, signifying better model performance and training stability. Unlike Inception Score (IS), FID accounts for both quality and diversity.