Adversarial Robustness

The measured accuracy of a model on a specific adversarial test set, typically generated by a known attack algorithm like Projected Gradient Descent (PGD). This is a practical, lower-bound estimate of resilience.

• Measurement: Percentage of adversarial examples correctly classified.
• Limitation: It only certifies robustness against the specific attack used; a stronger attack may break the defense.
• Standard Benchmark: Robustness against PGD with an L-infinity perturbation budget of 8/255 on CIFAR-10.

A formal, mathematical guarantee that a model's prediction will remain constant for any input perturbation within a defined Lp-norm radius. Unlike empirical robustness, this provides a provable lower bound.

• Mechanism: Techniques like Randomized Smoothing construct a smoothed classifier and use statistical hypothesis testing to certify the radius.
• Key Metric: The certified radius R around an input x where the prediction is provably invariant.
• Trade-off: Certified methods often achieve lower standard accuracy than empirical defenses.

The maximum allowed magnitude of an adversarial perturbation, defining the threat model's capacity. It constrains the attacker to ensure the modification remains imperceptible or physically plausible.

• L-infinity Norm: Limits the maximum change to any single pixel. A budget of ε=8/255 is standard for CIFAR-10.
• L2 Norm: Constrains the Euclidean distance of the perturbation vector.
• L0 Norm: Limits the total number of pixels an attacker can modify.
• Physical Constraints: Non-digital budgets may limit rotation, translation, or color shift.

The geometric property of the loss function in the input space around a data point. A sharp, highly curved loss landscape correlates with high vulnerability to adversarial perturbations.

• Visualization: Plotting the loss along a random direction vs. the adversarial gradient direction reveals sharp ravines in non-robust models.
• Adversarial Training Effect: This defense explicitly flattens the loss landscape, forcing the model to be locally constant around training points.
• Connection to Generalization: Flat minima in weight space are linked to better generalization; flatness in input space is linked to robustness.

A deceptive phenomenon where a defense causes the model's gradients to become useless (e.g., zero, saturated, or extremely noisy) to a gradient-based attacker. This creates a false sense of security.

• Obfuscated Gradients: The defense introduces non-differentiable operations or numerical instabilities.
• Detection: A defense is likely masking gradients if a black-box attack (which doesn't use gradients) is stronger than a white-box attack.
• Failure Mode: Gradient masking is not true robustness; attackers can bypass it by using a smooth surrogate model via Adversarial Example Transferability.

The empirically observed inverse relationship where increasing a model's adversarial robustness via methods like Adversarial Training degrades its performance on clean, unperturbed data.

• Hypothesized Cause: Robust features learned by the model may be fundamentally different from, and less useful for, standard classification features.
• Mitigation: Techniques like TRADES (TRadeoff-inspired Adversarial DEfense via Surrogate-loss) explicitly optimize a loss function that balances clean accuracy and robustness.
• Data Dependency: Larger, more complex datasets can partially alleviate this trade-off.

The property by which an adversarial example crafted to fool one specific surrogate model also causes misclassification in other independently trained target models with different architectures.

• Black-Box Exploit: Attackers train a local surrogate, generate attacks, and transfer them to the inaccessible target API.
• Cross-Architecture: Perturbations often transfer between CNNs, Vision Transformers, and even different task modalities.
• Defense Implication: A robust model must resist not only its own worst-case examples but also those generated by unknown external models.