Adversarial Training

Adversarial training is fundamentally framed as a min-max optimization problem. The inner maximization step generates adversarial examples that maximize the model's loss, while the outer minimization step updates model parameters to minimize loss on those examples.

• Objective: min_θ [ max_δ∈Δ L(f_θ(x+δ), y) ]
• The perturbation δ is constrained within a set Δ, typically an Lp-norm ball (e.g., L∞ epsilon-ball)
• This formulation was formalized by Madry et al. (2018) and remains the theoretical foundation for most adversarial training variants

The inner maximization step is most effectively solved using Projected Gradient Descent (PGD), an iterative attack that repeatedly perturbs the input and projects it back onto the allowed epsilon-ball.

• PGD-k: Uses k iterative steps with a small step size α
• Each step: x_t+1 = Π_{x+S}(x_t + α · sign(∇_x L(θ, x_t, y)))
• Models trained with PGD adversaries exhibit empirical robustness against a wide range of first-order attacks
• The computational cost scales linearly with the number of PGD steps, making it 3-10x more expensive than standard training

A central challenge in adversarial training is the trade-off between natural accuracy and robust accuracy. As a model becomes more robust to adversarial perturbations, its performance on clean, unperturbed data often degrades.

• Tsipras et al. (2019) demonstrated this trade-off is fundamental, not merely an artifact of optimization
• Robust features learned by the model may be less discriminative for clean examples
• Techniques like TRADES explicitly balance this trade-off by adding a regularization term that penalizes the divergence between clean and adversarial predictions
• In practice, a 5-10% drop in natural accuracy is common for highly robust models on datasets like CIFAR-10

Transferability—the property where adversarial examples crafted on one model fool another—plays a critical role in adversarial training. It enables black-box defense but also reveals limitations.

• Ensemble adversarial training: Augments data with adversarial examples generated from multiple surrogate models to improve robustness against unseen attacks
• However, adversaries can exploit transferability by crafting attacks on a held-out surrogate model not seen during training
• Obfuscated gradients, a phenomenon where defenses appear robust by masking gradient information, can give a false sense of security—models must be evaluated with adaptive attacks that circumvent gradient masking

While standard adversarial training focuses on Lp-norm constrained perturbations, real-world threats often involve transformations outside this model.

• Spatial transformations: Small rotations, translations, or warping that preserve perceptual similarity but fall outside epsilon-ball constraints
• Adversarial patches: Localized, unbounded perturbations (e.g., a sticker) that can be physically realized and placed in the real world
• Corruption robustness: Training on common corruptions like blur, noise, and weather effects improves resilience to distribution shift
• AugMix and similar techniques combine adversarial training with diverse data augmentations to broaden the robustness envelope

The primary barrier to widespread adoption of adversarial training is its significant computational overhead. Each training iteration requires generating adversarial examples, which itself requires multiple forward and backward passes.

• Single-step methods like FGSM-based training are faster but vulnerable to catastrophic overfitting, where robustness suddenly collapses
• Free adversarial training recycles gradients from the parameter update to simultaneously craft perturbations, reducing cost
• Fast adversarial training techniques aim to match PGD robustness with fewer inner steps
• For large-scale models like ImageNet classifiers, adversarial training can require weeks on hundreds of GPUs

Adversarial training requires generating powerful attacks during the training loop. The most common methods used to create these training adversaries include:

• Projected Gradient Descent (PGD): The de facto standard iterative white-box attack that maximizes loss within an Lp-norm epsilon-ball.
• Fast Gradient Sign Method (FGSM): A computationally cheaper, single-step attack often used for rapid adversarial example generation.
• Carlini & Wagner (C&W): An optimization-based attack that finds minimally distorted adversarial examples, useful for evaluating worst-case robustness.

The success of adversarial training is measured by specific metrics that quantify a model's resilience:

• Robust Accuracy: The primary metric, measuring classification accuracy on an adversarially perturbed test set.
• Attack Success Rate: The percentage of generated adversarial examples that successfully cause misclassification.
• Empirical Robustness: The model's measured resistance against a specific, finite set of attacks, as opposed to certified guarantees.

While adversarial training provides empirical robustness, certification provides mathematical guarantees that a prediction will not change for any input within a defined perturbation bound:

• Randomized Smoothing: A probabilistic certification technique that constructs a provably robust classifier by adding Gaussian noise and taking a majority vote.
• Formal Verification: Methods that use satisfiability modulo theories (SMT) or abstract interpretation to prove robustness properties of neural networks.

Standard adversarial training can degrade performance on clean data. Several variants address this trade-off:

• TRADES: Optimizes a loss function that explicitly balances the trade-off between natural accuracy and robust accuracy by regularizing the gap between clean and adversarial predictions.
• Interpolated Adversarial Training: Uses adversarial examples generated from interpolations between clean samples to improve generalization.

The effectiveness of adversarial training depends on the assumed threat model used during training:

• White-box Access: The attacker has full knowledge of model architecture and parameters. PGD-based training assumes this strongest threat.
• Black-box Access: The attacker can only query the model. Defenses trained only on white-box attacks may not transfer robustness to black-box settings.
• Transferability: Adversarial examples crafted for one model often fool others. Training against transferred attacks from an ensemble of models can improve generalization.

A related defensive technique that predates modern adversarial training. Defensive distillation trains a second model on the soft probability vectors (logits) of a previously trained model. This process smooths the loss surface, reducing the magnitude of adversarial gradients and making it harder for gradient-based attacks to find effective perturbations. It is often combined with adversarial training for layered defense.