Adversarial Training

Adversarial training is formally cast as a saddle point problem. The inner maximization generates the strongest possible adversarial examples, while the outer minimization adjusts model weights to classify them correctly.

• Objective: min_θ ρ(θ), where ρ(θ) = E_{(x,y)~D} [max_{δ∈S} L(θ, x+δ, y)]
• Inner Loop: Uses Projected Gradient Descent (PGD) to find the worst-case perturbation δ within the allowed threat model S.
• Outer Loop: Standard empirical risk minimization on the generated adversarial samples.

The defense is defined by the specific mathematical constraints of the adversary it simulates. A valid threat model must explicitly bound the perturbation.

• L∞-norm: Limits the maximum change to any single pixel. Common bound: ε = 8/255.
• L2-norm: Limits the Euclidean distance of the perturbation.
• Perturbation Budget: Defines the adversarial strength. A larger ε creates stronger attacks but may degrade clean accuracy.
• Attack Algorithm: Typically PGD with a fixed number of steps and step size.

A fundamental tension exists where increasing robustness to adversarial inputs often reduces performance on unperturbed, natural data.

• Robust Overfitting: A phenomenon where robust test accuracy degrades sharply after a certain number of training epochs, even while training loss continues to decrease.
• Data Complexity: The trade-off is more severe on complex datasets like ImageNet compared to simpler ones like CIFAR-10.
• Mitigation: Techniques like early stopping based on robust validation metrics and adding unlabeled data can partially close the gap.

Adversarial training is significantly more resource-intensive than standard training because it requires constructing adversarial examples on-the-fly for every batch.

• Cost Multiplier: A single PGD adversary with k steps increases training time by roughly a factor of k+1.
• Free Adversarial Training: An optimization that recycles gradients from the parameter update to compute perturbations, reducing the marginal cost to almost zero.
• Hardware Demand: State-of-the-art robust models often require training on high-memory GPUs for multiple days.

Unlike standard models that rely on brittle, high-frequency features, adversarially trained models learn robust features that align more closely with human perception.

• Saliency Maps: Robust models produce interpretable gradients that highlight the object of interest rather than background texture.
• Invertibility: The representations learned are more invertible, allowing direct reconstruction of the input image from the model's internal activations.
• Transfer Learning: Robust features have been shown to transfer better to downstream tasks like object detection and semantic segmentation.

Adversarial training is an empirical defense. It provides no mathematical guarantee of correctness against a novel, unseen attack.

• Gradient Masking Risk: A poorly implemented defense might cause obfuscated gradients, giving a false sense of security against gradient-based attacks but failing against black-box or adaptive attacks.
• Evaluation Standard: Robustness must be measured using a strong, standardized attack suite like AutoAttack.
• Certified Counterpart: For formal guarantees, adversarial training is often combined with Randomized Smoothing to provide a provable lower bound on the L2 radius.

The property by which adversarial examples crafted against one model also fool other independently trained models. This phenomenon enables black-box attacks where the adversary has no access to the target model's internals. Key insights:

• Surrogate model attacks: Train a substitute model on queried outputs, then generate transferable examples
• Ensemble methods: Crafting examples against multiple models simultaneously improves transfer rates
• Implications for defense: Transferability undermines security-by-obscurity; model architecture secrecy is not a reliable defense

Adversarial training reduces transferability by flattening the loss landscape, making perturbations less generalizable across models.