Adversarial Robustness

The maximum allowable magnitude of change an attacker can make to an input, typically defined by an Lp-norm constraint.

• L∞ (L-infinity): Caps the maximum change to any single pixel or feature. Common in image attacks.
• L2 (Euclidean): Limits the total squared difference across all features, allowing many small changes.
• L0: Restricts the number of pixels or features that can be altered. The choice of ε is critical; a model robust at ε=8/255 may collapse at ε=16/255.

The primary metric for adversarial robustness: the classification accuracy of a model when evaluated exclusively on adversarially perturbed inputs within a specific ε-budget.

• Clean Accuracy: Performance on unmodified test data.
• Robust Accuracy: Performance under worst-case perturbation. A common failure mode is gradient masking, where a defense appears to have high robust accuracy against weak attacks but collapses against adaptive attacks that circumvent the obfuscated gradients.

Instead of fixing a budget and measuring accuracy, this metric finds the smallest perturbation required to change the model's classification for a given input.

• Attacks like Carlini & Wagner (C&W) are explicitly designed to minimize this distortion.
• A higher average minimum perturbation across a dataset indicates a more robust model.
• This metric is intuitive but computationally expensive, as it requires solving an optimization problem for every sample.

A critical distinction in evaluating robustness claims:

• Empirical Robustness: Measured by testing against a specific set of attacks (e.g., PGD). A model may be empirically robust but vulnerable to a stronger, unseen attack.
• Certified Robustness: Provides a mathematical proof that no perturbation within a given Lp-norm ball can change the prediction. Techniques like randomized smoothing offer probabilistic guarantees. True resilience requires moving beyond empirical point estimates toward certified guarantees.

A fundamental tension observed by Tsipras et al. (2019): training for adversarial robustness often reduces clean accuracy.

• Robust models learn fundamentally different feature representations, often aligning more with human-perceptible shapes than brittle textural cues.
• This trade-off is not necessarily a law but a current empirical reality, driving research into architectures and training paradigms that can achieve both high clean and robust accuracy simultaneously.

The gold standard for assessing a defense. An adaptive attack is custom-designed with full knowledge of the defense mechanism to specifically circumvent it.

• If a defense uses feature squeezing, an adaptive attack incorporates the squeezing operation into the backward pass.
• If a defense uses randomness, an adaptive attack uses Expectation over Transformation (EOT) to optimize through the stochasticity. Any defense not evaluated against a rigorous adaptive attack cannot be considered robust.

The primary empirical defense for improving adversarial robustness. This technique augments the training dataset with adversarially perturbed examples—typically generated via Projected Gradient Descent (PGD)—labeled with their ground-truth class. By training on worst-case perturbations within an epsilon-ball, the model learns more robust decision boundaries. Key considerations:

• Computationally expensive, often requiring 3-10x more training time
• Provides robustness against the specific attack used during training
• May reduce accuracy on clean, unperturbed data

A false sense of security that occurs when a defense produces non-useful or zero gradients, preventing gradient-based attacks from optimizing. This is not true robustness. Common causes:

• Shattered gradients: Non-differentiable operations like input quantization or JPEG compression
• Stochastic gradients: Randomization that makes gradients noisy but still exploitable with Expectation Over Transformation (EOT)
• Vanishing gradients: Defenses that flatten the loss landscape near inputs Models exhibiting gradient masking remain vulnerable to black-box attacks and transfer attacks.

The gold standard for evaluating adversarial robustness claims. An adaptive attack assumes full knowledge of the defense mechanism and is specifically designed to circumvent it. For example, if a defense uses randomized input transformations, the adaptive attack applies Expectation Over Transformation (EOT) to compute gradients through the randomization. Key principle: no defense should be evaluated only against attacks it was designed to stop. Proper adaptive evaluation requires the attacker to optimize against the end-to-end defended model.

Extending robustness beyond digital pixels to the physical world introduces challenges from viewpoint variation, lighting changes, and sensor noise. Physical adversarial examples—such as stickers on stop signs or 3D-printed objects—must survive the imaging pipeline. The Expectation Over Transformation (EOT) framework optimizes perturbations over a distribution of real-world transformations. For embodied agents, robustness must account for multimodal sensor fusion and temporal consistency across video frames.