Adversarial Training

Adversarial training is fundamentally a min-max saddle point problem. The inner maximization generates the strongest possible adversarial example within an epsilon-ball constraint, while the outer minimization updates model weights to correctly classify that perturbed sample.

• Inner Loop: Projects perturbation in the direction of the loss gradient (e.g., via Projected Gradient Descent)
• Outer Loop: Standard empirical risk minimization on the generated adversarial examples
• Objective: Minimize the worst-case loss over the allowed perturbation set

The de facto standard for generating training-time adversaries is multi-step PGD. Unlike single-step methods like the Fast Gradient Sign Method (FGSM), PGD iteratively refines the perturbation with gradient steps and projection back onto the Lp-norm ball.

• Typically uses L∞ norm constraints with epsilon values like 8/255 for image data
• Random starts prevent the attacker from getting stuck in shallow loss maxima
• The number of steps (often 7–40) trades off attack strength against training compute cost

Unlike standard training, adversarial training exhibits a unique failure mode called robust overfitting, where test-time robust accuracy degrades sharply after a certain number of epochs while clean accuracy remains stable.

• Mitigated by early stopping based on a held-out adversarial validation set
• Cyclic learning rates and weight averaging (SWA) help stabilize convergence
• Indicates that memorizing training-set adversarial examples does not generalize to new attacks

A well-documented tension exists: models hardened with adversarial training typically suffer a degradation in clean data accuracy compared to undefended baselines. This is not a failure but a fundamental property of learning invariant, robust features.

• Robust models discard non-robust features that are predictive but brittle
• The gap widens on complex datasets like ImageNet
• TRADES loss explicitly balances the trade-off with a regularization parameter β

Adversarial training increases training time by a factor proportional to the number of PGD steps. A K-step PGD attacker requires K forward and backward passes per minibatch, making the technique 5–10× more expensive than standard training.

• Free adversarial training recycles gradients from the weight update to simultaneously craft perturbations, reducing overhead
• Single-step attacks (FGSM) are cheaper but vulnerable to catastrophic overfitting where robustness suddenly collapses to 0%
• Large-scale adversarial training on ImageNet requires distributed training across dozens of GPUs for days

A properly adversarially trained model produces gradients that are less transferable to surrogate models. Attackers crafting perturbations on a robust model find those perturbations fail against other architectures, indicating the defense has learned genuinely invariant features rather than engaging in gradient masking.

• Gradient obfuscation check: Verify that single-step attacks are weaker than multi-step attacks
• Black-box transfer test: Attacks generated on a held-out model should also fail
• Passing these checks distinguishes true robustness from gradient masking

An input perturbed with imperceptible noise that causes a model to misclassify with high confidence. These perturbations are crafted by maximizing the model's loss within a small Lp-norm bound (typically L∞ or L2). For instance, adding a carefully calculated pixel-level distortion to a panda image causes a classifier to see a gibbon with 99.3% confidence, while humans perceive no change.

The standard white-box attack used to generate adversarial examples during training. PGD iteratively:

• Takes small steps in the direction of the loss gradient
• Projects the perturbed input back onto an epsilon-ball around the original
• Repeats for 10-40 iterations

This multi-step variant is significantly stronger than the single-step Fast Gradient Sign Method (FGSM) and is the de facto benchmark for evaluating adversarial robustness.

A model's measured resilience to input perturbations, quantified as accuracy under attack. Key metrics include:

• Robust accuracy: Classification accuracy on adversarially perturbed test sets
• Minimum perturbation magnitude: The smallest Lp distortion required to flip a prediction
• Certified radius: A provable guarantee that no perturbation within a specified norm can change the output

Robustness typically trades off against clean accuracy—a phenomenon known as the robustness-accuracy trade-off.

A false sense of security where a defense produces non-useful or zero gradients, causing gradient-based attacks like PGD to fail during evaluation. This does not indicate true robustness. Common causes:

• Shattered gradients: Non-differentiable or randomized defenses
• Stochastic gradients: Defenses relying on randomness that attackers can bypass with Expectation Over Transformation (EOT)
• Exploding/vanishing gradients: Numerical instability in the loss landscape

Models exhibiting gradient masking remain vulnerable to black-box attacks and transfer attacks.

A formal mathematical guarantee that a model's prediction remains unchanged for any perturbation within a specified Lp-norm radius. Unlike empirical defenses that can be broken by stronger attacks, certified methods provide provable bounds. The leading technique is Randomized Smoothing:

• Adds Gaussian noise to inputs during inference
• Aggregates predictions over multiple noisy samples
• Uses statistical hypothesis testing to compute a certified L2 radius

This transforms any base classifier into a certifiably robust smoothed classifier.

The gold standard evaluation methodology that assumes the attacker has full knowledge of the defense mechanism and tailors the attack accordingly. An adaptive attack against adversarial training would:

• Use the exact same loss function and threat model used during training
• Account for any preprocessing or detection steps
• Optimize directly against the defended model's gradients

If a defense survives adaptive attacks, it demonstrates genuine robustness rather than gradient masking. Papers claiming robustness without adaptive evaluation are considered incomplete.