Adversarial Perturbation

The defining feature of an adversarial perturbation is its minimal magnitude. The perturbation is constrained by an Lp-norm budget (often L∞ or L2) to ensure the modified input remains visually or semantically identical to the original for a human observer, while completely altering the model's perception.

• L∞ Norm: Limits the maximum change to any single pixel, creating a uniform noise mask.
• L2 Norm: Restricts the Euclidean distance, allowing subtle, distributed changes across many pixels.
• Psychological Invisibility: The attacker exploits the gap between human visual processing and a model's high-dimensional feature extraction.

Perturbations crafted to fool a specific surrogate model (white-box access) often retain their malicious efficacy against a different, unknown target model (black-box access). This property makes real-world attacks viable without direct access to the production system.

• Decision Boundary Similarity: Models trained for similar tasks learn comparable, non-robust features that are universally exploitable.
• Ensemble-Based Attacks: Attackers generate perturbations against multiple surrogate models simultaneously to maximize the probability of transfer.
• Practical Impact: Enables attackers to deceive proprietary cloud APIs without stealing the underlying model architecture or weights.

Adversarial perturbations are not confined to the digital realm. Robust Physical Perturbations (RPP) maintain their attack integrity despite variations in lighting, angle, distance, and camera sensor noise.

• Expectation over Transformation (EOT): A technique that optimizes perturbations to remain effective across a distribution of physical transformations like rotation and blur.
• Adversarial Patches: Highly visible, localized perturbations that can be printed and placed in a scene to universally disable object detectors.
• Domain Gap: The primary challenge is ensuring the perturbation survives the analog-to-digital conversion of a camera pipeline.

The attacker's objective defines the perturbation's structure. A non-targeted attack simply requires the model to output any incorrect label, while a targeted attack forces the model to output a specific, attacker-chosen class.

• Non-Targeted: Maximizes the loss for the true class. Easier to compute and requires a smaller perturbation budget.
• Targeted: Minimizes the loss for the target class. This is a harder optimization problem, often requiring a larger perturbation.
• Source/Target Pairs: In facial recognition, an attacker might use a perturbation to impersonate a specific authorized individual (targeted) rather than just failing authentication.

Perturbations are generated by analyzing the model's loss gradient with respect to the input. Attackers exploit the linearity of deep networks in high-dimensional spaces, pushing the input in the direction that maximizes error.

• Fast Gradient Sign Method (FGSM): A one-step attack that uses the sign of the gradient to create a rapid perturbation.
• Projected Gradient Descent (PGD): An iterative, multi-step variant that applies FGSM repeatedly with a small step size and projects the result back into the epsilon-ball.
• Carlini & Wagner (C&W): A powerful optimization-based attack that formulates the perturbation as an objective minimizing both distortion and misclassification loss.

Beyond pixel-space noise, perturbations can be semantic—changing high-level attributes like color balance, texture, or even object rotation in a way that is natural to humans but catastrophic for the model.

• Hue Shifting: Subtly altering the color spectrum of an image without changing its content.
• Spatial Transformations: Applying a learned, smooth deformation field to warp the image geometry.
• Natural Adversarial Examples: Unmodified, naturally occurring images that models consistently misclassify due to out-of-distribution features, representing a semantic gap in the training data.

A foundational white-box attack that generates adversarial examples in a single step. FGSM computes the gradient of the loss function with respect to the input image, then applies a small perturbation in the direction that maximizes loss.

• Mechanism: x_adv = x + ε * sign(∇x J(θ, x, y))
• Key property: Speed over stealth—creates perturbations quickly but is easily defended against with adversarial training
• Origin: Introduced by Goodfellow et al. (2015) to demonstrate that linear behavior in high-dimensional spaces causes vulnerability

A defensive technique that injects adversarial examples into the training dataset to improve model robustness. The model learns to correctly classify both clean and perturbed inputs.

• Process: Generate adversarial examples on-the-fly during each training epoch, then minimize loss on both clean and adversarial samples
• Trade-off: Increases robustness but often reduces accuracy on clean data
• Min-max formulation: A saddle point problem where the inner maximization finds the strongest perturbation and the outer minimization trains against it

An inference-time attack where an adversary modifies a malicious sample to bypass a security classifier. Unlike poisoning, evasion does not alter the training process.

• Real-world example: Modifying malware binaries with benign code chunks to evade antivirus ML classifiers while preserving malicious functionality
• Contrast with poisoning: Evasion targets deployed models; poisoning corrupts the training pipeline
• Defense: Input sanitization, feature squeezing, and defensive distillation reduce evasion success rates

A formal guarantee that a model's prediction remains constant for any input within a mathematically defined perturbation radius. Unlike empirical defenses, certified robustness provides provable lower bounds.

• Techniques: Randomized smoothing adds Gaussian noise and uses statistical hypothesis testing to certify predictions
• Radius notation: A model is certified at radius r if no perturbation smaller than r can change the output
• Limitation: Current methods provide conservative bounds; the certified radius is often much smaller than the empirical robustness

A localized, highly visible perturbation placed in a physical scene to universally fool object detectors. Unlike imperceptible perturbations, patches are designed to work in the physical world.

• Physical realizability: Patches are printed and placed in the environment—e.g., a sticker on a stop sign causing misclassification
• Context independence: The patch is effective regardless of background, lighting, or viewing angle
• Defense challenges: Standard input sanitization fails because patches occupy a small pixel area with extreme values; specialized detectors or attention-based defenses are required

An intellectual property attack where an adversary steals model functionality by querying a black-box API and training a substitute model on the input-output pairs.

• Relationship to perturbation: Adversarial perturbations can be crafted against the extracted substitute model and transfer to the victim model
• Transferability: Perturbations generated on one model often fool others trained on similar tasks, making extraction a precursor to evasion
• Defense: Rate limiting, query monitoring, and output perturbation via differential privacy degrade extraction fidelity