Generative Adversarial Network (GAN)

The generator is a neural network that learns to transform random noise vectors into realistic synthetic data samples. Its objective is to produce outputs indistinguishable from real data.

• Input: A latent vector sampled from a Gaussian or uniform distribution
• Output: A synthetic data point (image, tabular row, time series)
• Training Signal: Feedback from the discriminator's classification
• Goal: Minimize the probability that the discriminator correctly identifies its outputs as fake

The generator never sees real data directly. It learns the data distribution solely through adversarial feedback, making it a powerful tool for privacy-preserving synthesis.

The discriminator is a binary classifier trained to distinguish real data samples from synthetic ones produced by the generator. It acts as an adaptive loss function.

• Input: A data sample, either real or generated
• Output: A probability score indicating whether the input is real
• Training Signal: Ground truth labels (real vs. fake)
• Goal: Maximize classification accuracy on both real and synthetic samples

As training progresses, the discriminator becomes increasingly sophisticated, forcing the generator to improve. This dynamic creates an arms race that drives both networks toward optimal performance.

GAN training is formulated as a minimax game between the generator (G) and discriminator (D). The objective function captures this competition:

• Generator's Objective: Minimize log(1 - D(G(z))), where z is random noise
• Discriminator's Objective: Maximize log(D(x)) + log(1 - D(G(z))), where x is real data
• Nash Equilibrium: The theoretical point where the generator perfectly replicates the data distribution and the discriminator can only guess at 50% accuracy

In practice, training is notoriously unstable. Common failure modes include mode collapse, vanishing gradients, and oscillatory behavior. Techniques like Wasserstein loss and gradient penalty help stabilize convergence.

Mode collapse is a catastrophic failure condition where the generator learns to produce only a narrow subset of the real data distribution, ignoring entire modes of variation.

• Symptom: The generator outputs highly similar or identical samples regardless of input noise
• Root Cause: The generator finds a single output that reliably fools the current discriminator and exploits it
• Impact: Synthetic data lacks diversity and fails to represent rare classes or edge cases

Mitigation strategies include minibatch discrimination, unrolled GANs, and Wasserstein GANs with gradient penalty. Detecting mode collapse requires monitoring sample diversity metrics during training.

Conditional GANs extend the standard architecture by feeding auxiliary information—such as class labels, attributes, or constraints—to both the generator and discriminator.

• Mechanism: Labels are concatenated with the noise vector (generator) and data sample (discriminator)
• Use Case: Generating synthetic data for a specific class or satisfying a constraint
• Example: CTGAN uses conditional vectors and training-by-sampling to handle imbalanced categorical columns in tabular data

Conditioning enables targeted data augmentation, fairness-aware synthesis, and scenario modeling where specific data characteristics are required.

The Wasserstein GAN replaces the standard discriminator with a critic that estimates the Earth Mover's Distance between real and generated distributions.

• Key Innovation: Uses Wasserstein distance instead of Jensen-Shannon divergence
• Benefits: More stable training, meaningful loss curves that correlate with sample quality, and reduced mode collapse
• Constraint: The critic must be 1-Lipschitz, enforced via weight clipping or gradient penalty (WGAN-GP)

WGANs provide a more theoretically grounded training objective. The critic's loss reliably indicates generation quality, unlike standard GAN loss which can be misleading.

A catastrophic training failure where the generator learns to produce only a single or very limited variety of output, ignoring the diversity of the real data distribution. Instead of generating varied handwritten digits, it might produce only a '3'. This happens when the generator finds a specific output that reliably fools the current discriminator and exploits it repeatedly.

• Unrolled GANs mitigate this by allowing the generator to anticipate the discriminator's future response.
• Minibatch discrimination allows the discriminator to look at entire batches of generated data to detect lack of variety.

Also known as the Earth Mover's Distance, this metric measures the minimum cost of transforming one probability distribution into another. In WGANs, it replaces standard cross-entropy loss to provide a smoother, more meaningful gradient signal to the generator, even when the real and generated distributions do not overlap.

• Solves vanishing gradients common in early GAN training.
• Requires the discriminator (called a critic) to be 1-Lipschitz continuous, often enforced via weight clipping or gradient penalty.

The adversarial 'detective' network in a GAN. Its objective is to accurately classify inputs as either real (drawn from the training set) or fake (produced by the generator). It outputs a probability score. The discriminator's loss function drives it to maximize the distance between its confidence on real data and its confidence on synthetic data.

• In a WGAN, the discriminator is replaced by a critic that outputs a real-valued score instead of a probability.
• A discriminator that becomes too powerful too quickly can completely stifle generator learning.

The 'forger' network that learns to map random noise vectors (latent space) to realistic synthetic data samples. It takes a latent vector z (sampled from a Gaussian or uniform distribution) and upsamples it through transposed convolutions or linear layers to produce an output matching the dimensions of the real data.

• The generator never sees the real data directly; it learns only through the gradients backpropagated from the discriminator's verdict.
• Conditional GANs (cGANs) feed a class label into the generator to control the type of output produced.

The minimax game framework where the generator and discriminator are trained simultaneously in a zero-sum competition. The generator minimizes log(1 - D(G(z))) while the discriminator maximizes log(D(x)) + log(1 - D(G(z))). This dynamic pushes the generator to produce samples indistinguishable from real data.

• Training is notoriously unstable; hyperparameter tuning is critical.
• Two time-scale update rule (TTUR) uses different learning rates for the generator and discriminator to stabilize convergence.

The process of generating synthetic data points that satisfy specific, user-defined constraints. In a Conditional GAN (cGAN), both the generator and discriminator are conditioned on auxiliary information y, such as a class label or a text embedding. This allows targeted generation of minority classes to fix imbalanced datasets.

• Enables data augmentation for specific edge cases.
• CTGAN applies this principle specifically to tabular data, conditioning on discrete columns to model complex non-Gaussian distributions.