Adaptive Discriminator Augmentation (ADA)

ADA continuously monitors the discriminator's behavior to detect overfitting without requiring a separate validation set. It tracks the gap between real and generated logits, using the statistic r_t = E[D_train] - E[D_validation]. When the discriminator memorizes the training set, this gap widens. ADA uses this signal to dynamically adjust augmentation probability p, applying stronger augmentations only when overfitting is detected, preserving training signal strength in data-rich regimes.

ADA applies a fixed set of augmentations to both real and generated samples before they enter the discriminator, ensuring the discriminator never sees a raw image. The standard pipeline includes:

• Pixel blitting: x-flips, 90-degree rotations, integer translation
• Geometric transformations: isotropic scaling, arbitrary rotation, anisotropic scaling
• Color transformations: brightness, contrast, luma flip, hue rotation, saturation
• Filtering: image-space filtering, additive Gaussian noise, cutout Each transformation is applied with probability p, where p is the adaptive parameter controlled by the overfitting heuristic.

The augmentation probability p is not binary. ADA adjusts p incrementally based on the overfitting metric:

• If r_t exceeds a threshold (indicating overfitting), p increases by a fixed step
• If r_t falls below the threshold, p decreases This creates a negative feedback loop that maintains the discriminator at the boundary of overfitting, maximizing its capacity utilization without memorization. The adjustment occurs every four minibatches, balancing responsiveness with training stability.

All augmentations in the ADA pipeline are designed to be invertible or probability-symmetric, ensuring the generator does not leak augmentation artifacts into generated images. Since the discriminator only sees augmented images, the generator receives gradients through augmented samples. If augmentations were non-invertible, the generator would learn to produce pre-augmented outputs. ADA avoids this by using transformations that preserve the underlying data distribution, maintaining the theoretical guarantees of the GAN training objective.

ADA enables stable GAN training on datasets as small as 1,000–5,000 images, where traditional GANs collapse. Key performance characteristics:

• On FFHQ with 5,000 images, ADA achieves FID scores comparable to StyleGAN2 trained on 70,000 images without augmentation
• Eliminates the need for differential augmentation or separate augmentation strategies for generator and discriminator
• Works as a drop-in replacement for standard discriminator training, requiring no architectural modifications This makes ADA critical for domains with inherent data scarcity, such as medical imaging, satellite imagery, and RF signal classification.

ADA was originally developed and validated within the StyleGAN2 framework but is architecture-agnostic. In the canonical implementation:

• Augmentations are applied inside the discriminator's forward pass before the first convolutional layer
• The augmentation probability p is initialized at 0.0 and adapts over the first few thousand kimg
• The overfitting heuristic uses a validation set of real images held out from the discriminator's training batch
• All hyperparameters (threshold, step size, min/max p) are standardized across datasets, requiring no per-dataset tuning

The primary failure mode that ADA prevents. When training data is scarce, the discriminator rapidly memorizes individual real samples rather than learning generalizable features of the data distribution. This causes the discriminator's confidence on real samples to saturate near 1.0, providing zero meaningful gradient to the generator and halting training.

• Symptom: Discriminator loss collapses to near zero
• Consequence: Generator receives no useful feedback
• ADA Solution: Dynamically augments both real and fake samples to prevent memorization

ADA introduces a tunable hyperparameter that controls the probability of applying a stochastic augmentation pipeline to discriminator inputs. The key innovation is adaptive scheduling: the probability is adjusted dynamically based on the degree of overfitting detected.

• Heuristic: Monitors the sign of the discriminator output on real samples
• Overfitting Signal: If E[D(x)] > 0.9, increase augmentation probability
• Stabilization: Reduces p as the discriminator becomes properly regularized

ADA applies a carefully designed set of transformations that preserve the semantic content of the signal while preventing exact memorization. For RF data, this includes domain-appropriate augmentations applied to both real and generated samples before they enter the discriminator.

• Pixel Blitting: Translations, flips, and rotations for spectrogram inputs
• Geometric Transformations: Scaling, rotation, and anisotropic scaling
• Color/Intensity: Brightness, contrast, and hue shifts
• RF-Specific: IQ imbalance, phase rotation, and additive Gaussian noise

A catastrophic GAN failure where the generator produces only a narrow subset of the target distribution. In RF contexts, this means generating only one modulation scheme or a single channel condition repeatedly. ADA mitigates mode collapse indirectly by preventing discriminator overfitting, which is a known contributor to the problem.

• Symptom: Generator outputs lack diversity
• RF Example: Generating only QPSK signals when trained on mixed modulation dataset
• Related Solution: Wasserstein GAN (WGAN) with gradient penalty

The formal name for the ADA technique, introduced by Karras et al. in 2020. Unlike standard data augmentation applied only to the training set, ADA applies augmentations to all discriminator inputs—both real and generated—with a probability p that adapts during training. This prevents the discriminator from learning augmentation artifacts as distinguishing features.

• Key Insight: Augmenting only real samples teaches the discriminator to detect augmentation
• Balanced Approach: Identical augmentation pipeline for real and fake inputs
• Leakage Prevention: Ensures augmentations do not leak into the generator via the adversarial loss