Adversarial Example

The defining characteristic of an adversarial example is that the perturbation is imperceptible to a human observer while completely altering the model's prediction. This is enforced by an Lp-norm perturbation budget—typically L-infinity—which caps the maximum pixel-level change. The attack must remain within this imperceptibility threshold to be considered stealthy.

• L-infinity norm: Limits the maximum change to any single pixel (e.g., ±8/255).
• L2 norm: Constrains the Euclidean distance of the perturbation vector.
• L0 norm: Restricts the total number of pixels that can be altered.

Adversarial examples do not simply cause a model to guess incorrectly; they force a high-confidence misclassification to an attacker-chosen target class. The model outputs a probability distribution that is sharply peaked on the wrong label, often with >99% confidence. This reveals that the vulnerability is not a boundary case but a structural blind spot in the learned feature space.

• Targeted attack: Forces classification to a specific incorrect class.
• Untargeted attack: Causes any misclassification away from the true label.
• Confidence gap: The model's certainty on the adversarial input often exceeds its certainty on clean inputs.

A critical and dangerous property: an adversarial example crafted to fool Model A will often fool Model B, even if Model B has a different architecture or was trained on a disjoint dataset. This adversarial example transferability enables black-box attacks where the attacker has no direct access to the target model's weights or gradients.

• Intra-architecture transfer: Examples transfer between models with the same design.
• Inter-architecture transfer: Examples crafted on a CNN fool a Vision Transformer.
• Attack proxy: Attackers train a local substitute model to generate transferable perturbations.

Adversarial examples arise from the linear nature of neural networks in high-dimensional space. The decision boundary is not smooth; it contains sharp, non-robust features that are highly sensitive to small input variations. Loss landscape visualization reveals steep cliffs where a single gradient step can cross the boundary. This is a consequence of the model learning spurious correlations rather than robust, generalizable features.

• Sharp curvature: High local Lipschitz constant near data points.
• Non-robust features: Patterns predictive on the training distribution but meaningless to humans.
• Boundary tilting: The decision surface is nearly orthogonal to the perturbation direction.

Adversarial examples are not confined to the digital domain. Physical adversarial attacks demonstrate that perturbations can survive printing, camera capture, and varying lighting conditions to fool real-world systems. An adversarial patch—a localized, highly visible sticker—can cause a stop sign to be classified as a speed limit sign, posing a direct threat to autonomous vehicles and surveillance systems.

• Robust physical perturbations: Designed to survive the image capture pipeline.
• Adversarial patch attacks: Localized perturbations that dominate the model's attention.
• Environmental invariance: Effective across angles, distances, and lighting conditions.

The most efficient adversarial examples are constructed by exploiting the model's own loss gradient. The Fast Gradient Sign Method (FGSM) takes a single step in the direction that maximizes the loss. More powerful iterative methods like Projected Gradient Descent (PGD) apply multiple small steps, projecting back onto the epsilon-ball after each iteration to find the strongest perturbation within the budget.

• FGSM: Single-step, computationally cheap, lower attack success.
• PGD: Multi-step, gold-standard for evaluating empirical robustness.
• C&W Attack: Optimization-based, finds minimal-distortion perturbations.

An attack vector executed at test time where an adversary modifies a malicious input sample to cause misclassification. Unlike poisoning attacks that corrupt training data, evasion attacks exploit the model's learned decision boundary directly. The attacker crafts a perturbation—often constrained by an Lp-norm budget—that flips the label while remaining imperceptible. This is the primary delivery mechanism for an adversarial example in production systems.

A defensive technique that injects adversarial examples into the training dataset with their correct labels. By exposing the model to worst-case perturbations during learning, the optimizer flattens the loss landscape around data points. This is widely considered the most empirically robust defense against Projected Gradient Descent (PGD) attacks, though it incurs a computational cost and can slightly reduce accuracy on clean data.

A phenomenon where a defense causes the model's gradients to become useless or misleading to an attacker, creating a false sense of security. This often occurs when a model is non-differentiable or saturates its outputs, preventing gradient-based attacks from finding effective perturbations. Security engineers must distinguish true robustness from gradient masking, as black-box attacks or transfer-based attacks can easily bypass these obfuscated gradients.

A formal, verifiable guarantee that a model's prediction will remain constant for any perturbation within a specified Lp-norm radius. Unlike empirical defenses that can be broken by a stronger attack, certified defenses provide a provable lower bound on robustness. Techniques like Randomized Smoothing construct a smoothed classifier by adding Gaussian noise and aggregating predictions, yielding a statistical certificate against L2-norm adversaries.

The property by which an adversarial example crafted to fool one model also causes misclassification in other independently trained models. This cross-model vulnerability enables black-box attacks: an adversary trains a local surrogate model, generates white-box attacks against it, and deploys them against a remote target API. Transferability is driven by shared geometric features in the decision boundaries learned from similar data distributions.

An attack that remains effective after being printed and captured by a camera in the physical world. This demonstrates that adversarial vulnerabilities are not merely digital artifacts. Researchers have shown that adversarial patches—localized, highly visible perturbations—can cause a stop sign classifier to read 'Speed Limit 45' or make a person invisible to a detector, posing real-world risks for autonomous systems.