Evasion Attack

An evasion attack is distinguished by its execution after a model is deployed. The adversary crafts a malicious input—an adversarial example—specifically designed to be misclassified by the live model during prediction. This contrasts with training-time attacks like data poisoning, which corrupt the model during its learning phase. The attack exploits the gap between the model's training distribution and the potentially infinite space of possible inputs at runtime.

The core mechanism involves adding a subtle, often imperceptible, perturbation to a legitimate input. This perturbation is not random; it is calculated to maximally confuse the model. Common methods include:

• Fast Gradient Sign Method (FGSM): A single-step attack using the model's gradient.
• Projected Gradient Descent (PGD): A stronger, iterative variant of FGSM.
• Carlini & Wagner (C&W): An optimization-based attack that finds minimal perturbations. The goal is to create an input that appears normal to a human but lies on the wrong side of the model's decision boundary.

Evasion attacks are categorized by the attacker's assumed knowledge of the target model:

• White-Box Attacks: The attacker has full access to the model's architecture, parameters, and gradients. This is the most powerful setting, used for security evaluation (e.g., red-teaming).
• Black-Box Attacks: The attacker only has query access to the model's input-output API. Attacks often rely on transferability, where an example crafted on a surrogate model fools the target, or query-based strategies to estimate gradients.
• Gray-Box Attacks: A hybrid scenario with partial knowledge, such as knowing the model architecture but not its trained weights.

The adversary's goal defines the attack's precision:

• Untargeted Attack: The objective is simply to cause any misclassification. For an image classifier, this means changing "cat" to anything other than "cat." This is often easier to achieve.
• Targeted Attack: The objective is to cause the model to output a specific, incorrect class. For example, forcing a malware detector to classify malicious code as "benign," or making an autonomous vehicle's vision system misread a stop sign as a speed limit sign. Targeted attacks require more sophisticated crafting.

Evasion attacks are not confined to digital inputs. Physical adversarial attacks apply perturbations to real-world objects. Key examples include:

• Patch Attacks: Applying a visible, often colorful sticker (an adversarial patch) to an object (e.g., a stop sign) to cause misclassification.
• Object Perturbation: Subtly altering the texture or shape of an object. These attacks are critical for evaluating the robustness of systems like autonomous vehicles, facial recognition, and robotics, where sensors interact directly with the physical environment.

Building adversarial robustness requires specific defensive strategies, as standard training offers little protection. Primary methods include:

• Adversarial Training: Retraining the model on a mixture of clean and adversarial examples, fundamentally hardening its decision boundaries. PGD-based adversarial training is a standard benchmark.
• Input Transformation & Detection: Preprocessing inputs to remove potential perturbations (e.g., via compression or denoising) or using a separate detector to flag adversarial examples before they reach the main model.
• Randomized Smoothing: A provable defense that certifies a model's prediction within a radius of the input, guaranteeing robustness to bounded perturbations.

The most studied domain for evasion, where imperceptible pixel-level perturbations cause dramatic misclassification. Common techniques include:

• Fast Gradient Sign Method (FGSM): A single-step attack using the sign of the loss gradient to create perturbations.
• Projected Gradient Descent (PGD): A stronger, iterative variant of FGSM that is a standard benchmark for adversarial robustness.
• Carlini & Wagner (C&W): An optimization-based attack designed to find minimal perturbations, often used to break defensive distillation. Real-world implications include fooling facial recognition systems or causing autonomous vehicles to misread traffic signs.

Evasion against text models involves semantically preserving but adversarially crafted inputs. Key vectors include:

• Adversarial Typographical Errors: Introducing character-level swaps, insertions, or deletions (e.g., 'wireless' to 'wir3less') to bypass spam or toxicity detectors.
• Synonym Substitution: Replacing words with contextually similar alternatives from an embedding space to preserve meaning but alter model output.
• Semantic Perturbation: Adding distracting or contradictory sentences to manipulate sentiment analysis or classification. These attacks challenge models' robustness to distributional shifts in natural language.

A critical security application where attackers modify malicious software or network packets to avoid ML-based detection systems.

• Payload Obfuscation: Adding benign, redundant code sections or encrypting parts of malware to alter its feature signature without changing core functionality.
• Header Manipulation: Slightly modifying packet header fields (e.g., TTL, window size) to evade anomaly-based intrusion detection systems (IDS).
• Format Exploits: Using different file formats or encodings that parsers handle incorrectly, causing feature extractors to miss malicious content. This creates a continuous arms race between detector updates and adversarial sample generation.

Attacks where perturbations are applied to objects in the real world, posing direct risks to cyber-physical systems.

• Patch Attacks: Applying a visible, often colorful sticker or patch to an object (e.g., a stop sign) to cause misclassification by a vision system.
• Camouflage: Designing clothing or car wraps with patterns that confuse person or vehicle detectors.
• 3D Adversarial Objects: Printing objects with textures or shapes calculated to be misclassified from multiple viewpoints. These attacks are particularly concerning for autonomous vehicles, surveillance, and robotics, as they bypass digital-only defenses.

Crafting audio perturbations that are inaudible or perceived as background noise to humans but cause transcription errors in Automatic Speech Recognition (ASR) systems.

• Over-the-Air Attacks: Playing specially crafted audio that commands a voice assistant (e.g., to unlock a door or make a purchase) without the user's consent.
• Adversarial Music: Embedding hidden commands within music tracks or white noise.
• Phoneme Manipulation: Slightly altering the pronunciation of specific phonemes to change the transcribed text. These attacks exploit the disconnect between human and machine perception of audio signals.

A practical attack vector where the adversary has no internal model knowledge, relying solely on input-output queries to craft evasive samples.

• Score-Based Attacks: Using the confidence scores (probabilities) returned by the model to estimate gradients and perform iterative optimization.
• Decision-Based Attacks: Using only the final predicted label (hard decision) to perform boundary searches, such as the Boundary Attack.
• Transfer Attacks: Crafting an adversarial example on a locally trained surrogate model, then hoping it transfers to the unknown target model. This approach is highly relevant for attacking proprietary models accessed via APIs, where internal weights are hidden.

Adversarial robustness is the property of a machine learning model that quantifies its ability to maintain correct predictions when subjected to adversarial attacks. It is measured by robust accuracy, which is the classification accuracy on a test set containing adversarial examples. A model with high adversarial robustness is more reliable and secure in production environments where evasion attempts are possible.

• Key Metric: Robust accuracy versus standard accuracy.
• Goal: To minimize the performance gap between clean and adversarial inputs.
• Challenge: Often involves a trade-off with standard accuracy on benign data.

Adversarial training is the primary defensive technique used to improve a model's adversarial robustness. It involves augmenting the standard training dataset with adversarial examples generated on-the-fly during training. This forces the model to learn a more generalized and resilient decision boundary.

• Process: Iteratively generates attacks (e.g., using Projected Gradient Descent) and includes them as training data.
• Outcome: The model learns to be invariant to small, malicious perturbations.
• Consideration: Computationally expensive and can sometimes lead to gradient masking, a false sense of security.

These terms define the attacker's level of knowledge about the target model, which dictates the attack strategy.

• White-Box Attack: The attacker has full access to the model's architecture, parameters, and gradients. Methods like Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) are white-box. They are highly effective for evaluating worst-case robustness.
• Black-Box Attack: The attacker has no internal knowledge, only query access to the model's input-output behavior. Attacks are often query-based or rely on transfer attacks from a surrogate model. This scenario is more realistic for many deployed APIs.

A physical adversarial attack is an evasion attack executed in the physical world, where perturbations are applied to real-world objects. These attacks target computer vision systems like those in autonomous vehicles or facial recognition.

• Key Characteristic: The adversarial perturbation must remain effective under varying viewpoints, lighting, and camera noise.
• Common Technique: Patch attacks, where a visible, often semantically meaningful sticker is placed on an object (e.g., causing a stop sign to be misclassified).
• Defense Challenge: Requires robustness to a much wider distribution of transformations than digital attacks.

Projected Gradient Descent is a powerful, iterative white-box attack algorithm and the cornerstone for modern adversarial training. It is considered a universal first-order adversary.

• Mechanism: Applies the Fast Gradient Sign Method (FGSM) multiple times with a small step size. After each step, the perturbation is projected back onto a valid norm-ball (e.g., L∞) to ensure it stays within the allowed threat model.
• Use Case: The standard benchmark for evaluating adversarial robustness. If a model is robust to PGD, it is generally robust to other first-order attacks.
• Strength: Generates strong, high-confidence adversarial examples.

In AI security, red-teaming is the systematic, proactive practice of simulating adversarial attacks against a model or system to identify vulnerabilities before deployment. For evasion attacks, this involves dedicated teams crafting and testing a wide variety of adversarial examples.

• Objective: To discover failure modes, assess real-world robustness, and inform the development of defensive measures.
• Scope: Goes beyond automated attacks to include manual, creative exploitation of model weaknesses.
• Outcome: A comprehensive security assessment that drives improvements in model hardening and monitoring.