Adversarial Example

The defining feature of an adversarial example is that the perturbation is semantically meaningless to a human observer. A human would confidently assign the original label, while the model assigns a completely different one with high confidence.

• Lp-norm constraints: Attackers typically bound the perturbation using L0, L2, or L∞ norms to ensure visual fidelity.
• Just Noticeable Difference (JND): Effective attacks operate below the threshold of human perceptual awareness.
• Example: An image of a panda with an imperceptible noise layer is classified as a gibbon by a convolutional neural network with >99% confidence.

Adversarial examples crafted to fool one model often fool other models trained independently, even with different architectures or disjoint training datasets. This property is known as transferability.

• Black-box exploitation: An attacker can train a local substitute model, generate adversarial examples against it, and deploy them against a remote target API without any internal access.
• Decision boundary similarity: Transferability arises because independently trained models learn similar decision boundaries in high-dimensional space.
• Practical impact: This property makes black-box attacks viable against commercial machine-learning-as-a-service platforms.

Adversarial examples do not simply push a model into ambiguity; they cause high-confidence errors. The model assigns a probability near 100% to the incorrect class.

• Overconfidence exploitation: Attackers exploit the fact that softmax outputs saturate quickly in deep networks, creating large regions of high-confidence predictions far from the training data manifold.
• Logit-space manipulation: Small input perturbations are amplified through successive layers, causing dramatic shifts in the final logit values.
• Example: A stop sign with small stickers is classified as a speed limit sign with 99.8% confidence, a critical failure mode for autonomous vehicles.

Adversarial vulnerability is not a bug but a structural consequence of high-dimensional linearity. In a high-dimensional input space, a small perturbation per pixel accumulates into a large total change in the model's activation.

• Linear behavior hypothesis: Goodfellow et al. demonstrated that even linear models in high dimensions are vulnerable to adversarial perturbation.
• Local linearity: Deep networks, despite being globally non-linear, behave in a largely linear fashion locally, making them susceptible to gradient-based attacks.
• Curse of dimensionality: The volume of the input space grows exponentially with dimension, meaning the model's training data covers an infinitesimally small fraction of possible inputs.

Adversarial examples are defined relative to a specific threat model that specifies the perturbation budget and distance metric. The choice of norm fundamentally shapes the attack's characteristics.

• L∞-bounded attacks: Constrain the maximum per-pixel change, producing uniform, imperceptible noise (e.g., PGD with ε=8/255).
• L2-bounded attacks: Constrain the Euclidean distance, producing perturbations spread across many pixels.
• L0-bounded attacks: Constrain the number of pixels changed, producing sparse but potentially visible perturbations (e.g., the one-pixel attack).
• Non-norm constraints: Real-world attacks may use spatial transformations, rotations, or physical artifacts (adversarial patches) that violate pixel-space norm bounds.

Adversarial examples exploit non-robust features—patterns in the data that are predictive for classification but are incomprehensible and brittle to humans. Ilyas et al. demonstrated that models can achieve high accuracy relying solely on these features.

• Robust vs. non-robust features: Robust features remain predictive under adversarial perturbation; non-robust features flip their correlation.
• Dataset distillation: It is possible to construct a dataset where every input appears to be pure noise to humans, yet a model trained on it achieves non-trivial test accuracy on the original distribution.
• Defense implication: Adversarial training works by forcing the model to ignore non-robust features and rely on human-aligned, robust features.