Adversarial Robustness

The Fast Gradient Sign Method (FGSM) is a white-box attack that generates adversarial examples by calculating the gradient of the model's loss function with respect to the input. It perturbs the input in the direction that maximizes the loss, using a single step scaled by a small epsilon (ε).

• Mechanism: x_adv = x + ε * sign(∇_x J(θ, x, y))
• Key Trait: Computationally efficient, often used for adversarial training.
• Example: Adding imperceptible noise to a panda image causes a model to classify it as a gibbon with high confidence.

Projected Gradient Descent (PGD) is considered a universal first-order adversary. It is an iterative, multi-step extension of FGSM that performs gradient ascent to maximize loss, then projects the perturbed input back onto an ε-sized L∞ norm ball around the original input after each step.

• Mechanism: Iterative application: x_{t+1} = Proj_{x+ε}(x_t + α * sign(∇_x J(θ, x_t, y)))
• Key Trait: A strong baseline attack for evaluating robustness; often the benchmark for adversarial training.
• Defense: Models robust to PGD are generally more resilient to other first-order attacks.

The Carlini & Wagner (C&W) attack is a powerful optimization-based white-box attack designed to defeat defensive distillation and other early robustness techniques. It formulates attack generation as an optimization problem, minimizing perturbation while ensuring misclassification.

• Mechanism: Minimizes a custom loss: ||δ||_p + c * f(x+δ), where f is a function that is negative only for misclassification.
• Key Trait: Highly effective, often used as a stress test for defenses. It can target specific incorrect classes (targeted attack).
• Trade-off: Computationally expensive compared to FGSM or PGD.

The Jacobian-based Saliency Map Attack (JSMA) is a targeted attack that perturbs a minimal number of input features (e.g., pixels). It uses the model's Jacobian matrix to construct a saliency map, identifying which features most efficiently push the model toward a specific target class.

• Mechanism: Iteratively selects and modifies the most salient pixel(s) using forward derivatives.
• Key Trait: Creates sparse perturbations (changes few pixels), making it an L0-norm attack.
• Use Case: Demonstrates that attacks don't require small changes to all features; localized, sparse noise can be equally effective.

DeepFool is an efficient untargeted attack that finds the smallest perturbation needed to cross the model's decision boundary. It treats the classifier as linear in local regions and iteratively pushes the input toward the nearest boundary.

• Mechanism: Approximates the decision boundary with a hyperplane and projects the point onto it, repeating until misclassification.
• Key Trait: Often produces smaller perturbations (lower norm) than FGSM for the same misclassification result.
• Utility: Used to measure a model's robustness by calculating the average perturbation distance to the decision boundary across a dataset.

Universal Adversarial Perturbations are single, input-agnostic noise patterns that, when added to any natural image, cause misclassification with high probability for a given model. They exploit geometric correlations in the model's decision boundaries.

• Mechanism: Generated by aggregating minimal perturbations required to fool the model on a dataset, ensuring the cumulative perturbation remains small.
• Key Trait: Data-agnostic and model-specific. A single perturbation works across many inputs.
• Implication: Reveals a systemic vulnerability, not just an issue with individual samples. Defenses must harden the entire decision surface.

Adversarial examples are inputs to a machine learning model that have been intentionally perturbed in a way that is often imperceptible to humans but causes the model to make a high-confidence error. They are the primary weapon in testing and exploiting model robustness.

• Characteristics: Often crafted using gradient-based methods like the Fast Gradient Sign Method (FGSM) or Projected Gradient Descent (PGD).
• Purpose: Used for red-teaming models to evaluate weaknesses and for adversarial training to improve robustness.
• Example: Slightly modifying pixel values in an image of a panda causes a classifier to confidently label it as a gibbon.

Adversarial training is a defensive technique where a model is trained on a mixture of clean data and adversarially perturbed examples. This forces the model to learn a more robust decision boundary that is resistant to small input variations.

• Process: During training, adversarial examples are generated on-the-fly for each batch using an attack algorithm.
• Trade-off: Often improves robustness at a potential cost to standard accuracy on clean data (the robustness-accuracy trade-off).
• Foundation: A core method for building empirically robust models, forming the basis for many advanced defenses.

Gradient masking (or gradient obfuscation) is a phenomenon where a model appears robust because it produces unusable or misleading gradients, but its underlying decision boundary remains vulnerable. It creates a false sense of security.

• Causes: Can be induced by non-differentiable operations, shattered gradients, or stochastic defenses.
• Problem: Makes gradient-based white-box attacks (which rely on model gradients) fail, while the model remains susceptible to black-box attacks or gradient-free methods.
• Detection: A key sign is a large discrepancy between robustness to white-box and black-box attacks.

Certified robustness provides a mathematical guarantee that a model's prediction will not change within a defined region around an input (e.g., an L-p norm ball). Unlike empirical defenses, it offers provable security bounds.

• Methods: Includes Interval Bound Propagation (IBP) and Randomized Smoothing, which certifies a model by adding noise to inputs.
• Output: For a given input and perturbation budget (ε), the method certifies the prediction is stable or abstains.
• Use Case: Critical for high-assurance applications in finance, aviation, or healthcare where failure cannot be risked.

Input sanitization is a pre-processing defense that attempts to detect and remove or neutralize adversarial perturbations from an input before it is passed to the core model. It acts as a protective filter.

• Techniques: Includes feature squeezing (reducing color bit depth), JPEG compression, spatial smoothing, or using a denoising autoencoder.
• Advantage: Often model-agnostic and can be added to existing systems.
• Limitation: Can be circumvented by attackers who adapt their methods to bypass the specific sanitization function, leading to an arms race.

An adversarial patch is a localized, often visible perturbation applied to a physical object or digital image to cause targeted misclassification. Unlike subtle pixel-wide noise, patches are designed to be effective regardless of location or background.

• Characteristic: The attack is spatially constrained but not perturbation-constrained.
• Real-World Threat: Used in physical-world attacks, such as a sticker on a stop sign causing an autonomous vehicle's vision system to misread it.
• Defense: Particularly challenging, requiring robust object detection and segmentation models that can ignore out-of-context visual elements.