Critic Network

Unlike a standard discriminator that outputs a probability (e.g., 0 for fake, 1 for real), a critic outputs an unbounded scalar value. This score is an estimate of how much "work" (in terms of probability mass) is required to transform the generated distribution into the real one. Higher scores indicate the generated data is more 'real-like' according to the critic's current estimate of the Wasserstein distance.

• Key Implication: The critic's output is not a probability and does not use a sigmoid activation in its final layer.

For the Wasserstein distance estimate to be valid, the critic function must be 1-Lipschitz continuous, meaning its gradient magnitude must be bounded. WGANs enforce this constraint to ensure stable training. The primary method is gradient penalty (WGAN-GP), which adds a regularization term to the loss that penalizes the critic if the gradient norm deviates from 1.

• Alternative Method: The original WGAN used weight clipping, but this often leads to capacity underuse and optimization difficulties.

The critic is trained to maximize the difference between its scores for real and generated data. Its loss function is: L_critic = E[D(x_fake)] - E[D(x_real)] where D is the critic network. The critic is trained to maximize this difference (making real scores high and fake scores low). Concurrently, the generator is trained to minimize E[D(x_fake)], pushing its scores higher. This setup avoids the vanishing gradients common in standard GANs, as the critic provides a more linear, meaningful gradient signal.

Because the critic estimates the Wasserstein distance, the value of its loss (E[D(x_fake)] - E[D(x_real)]) correlates with generation quality. As training progresses, this value (typically negative) should generally decrease (become less negative), indicating the distributions are converging.

• Key Benefit: This provides a meaningful, interpretable metric to monitor during training, unlike the oscillating, uninterpretable loss of a standard GAN discriminator. It helps diagnose issues like mode collapse or training divergence.

A critic network is typically a feedforward or convolutional neural network similar in architecture to a standard discriminator, but without final sigmoid/softmax layers. It often requires sufficient capacity to model complex distributions. Due to the Lipschitz constraint, architectural choices like spectral normalization can be applied to each layer to directly enforce the constraint, sometimes replacing the need for a gradient penalty term and leading to more stable training dynamics.

The critic solves key problems of the standard GAN discriminator:

• Gradient Stability: Provides more reliable gradients, as it does not saturate when samples are easily classified.
• Mode Coverage: The Wasserstein loss is theoretically better at encouraging the generator to cover all modes of the data distribution, mitigating mode collapse.
• Training Balance: Less sensitive to the precise balance between generator and critic updates, though the critic is usually trained for more iterations per generator step (e.g., 5:1) to ensure a good Wasserstein estimate.

In a standard Generative Adversarial Network (GAN), the discriminator network is a neural network classifier trained to output a probability that a given sample is real (from the training data) versus fake (from the generator). Its objective is the opposite of the critic's: it performs binary classification, whereas the critic outputs an unbounded scalar score estimating distribution distance.

The Wasserstein Distance, or Earth Mover's Distance, is a metric that measures the minimum "cost" of transforming one probability distribution into another. It provides a continuous and meaningful gradient even when distributions have little overlap. The critic network in a WGAN is trained to approximate this distance, which is why its output is a score, not a probability.

• Key Property: Provides smoother gradients than the Jensen-Shannon divergence used in standard GANs.
• Result: Leads to more stable training and a loss value that correlates with generation quality.

A Lipschitz constraint is a mathematical condition that limits how fast a function's output can change relative to its input. In WGANs, the critic network must be a 1-Lipschitz function for the Wasserstein distance calculation to be valid. This constraint is enforced to prevent the critic from providing arbitrarily large gradients, which would destabilize training.

Common enforcement techniques include:

• Weight Clipping: An early method that clamps the critic's weights to a small fixed range.
• Gradient Penalty (WGAN-GP): A more advanced method that adds a penalty term to the loss based on the gradient norm, encouraging it to be close to 1.

Training stability refers to the challenge of maintaining a balanced, non-diverging dynamic between the generator and its adversary (discriminator/critic) throughout optimization. Standard GANs are notoriously unstable due to issues like vanishing gradients. The WGAN framework, with its critic network, was designed explicitly to improve stability.

• Critic's Role: By approximating a continuous distance metric, the critic provides more reliable gradients to the generator, even when the generated distribution is poor.
• Indicator: The critic's loss (the approximated Wasserstein distance) tends to correlate with sample quality, serving as a useful training metric.

Adversarial loss is the objective function that formalizes the competition in a GAN. For a WGAN, the adversarial loss is based on the Wasserstein distance.

• Critic's Loss: Maximize the score for real data and minimize the score for generated data: L_critic = E[critic(fake)] - E[critic(real)].
• Generator's Loss: Minimize the critic's score for its generated data: L_gen = -E[critic(fake)].
• Contrast: This differs from the standard GAN's minimax game using binary cross-entropy, where the discriminator's outputs can saturate, leading to weak generator gradients.

The generator network is the other half of the adversarial pair. It learns to map random noise from a latent space to synthetic data that mimics the real data distribution. In a WGAN, the generator is trained to minimize the output score of the critic network.

• Dynamic: The generator and critic are trained in alternation. The critic must be trained to optimality (or near-optimality) multiple times for each generator update to ensure it provides a good gradient estimate of the Wasserstein distance.
• Goal: To produce data such that the critic cannot differentiate its distribution from the real one, meaning the approximated Wasserstein distance approaches zero.