Attack Success Rate

Attack Success Rate is calculated as the percentage of generated adversarial examples that successfully cause the target model to produce an incorrect output. The fundamental formula is:

• ASR = (Number of Successful Attacks / Total Number of Attack Attempts) × 100%

A successful attack is defined by the attacker's objective:

• Untargeted Attack: The model outputs any class other than the ground truth
• Targeted Attack: The model outputs a specific, pre-chosen target class
• LLM Jailbreak: The model complies with a harmful request it would normally refuse

An ASR of 95% indicates near-complete vulnerability, while an ASR of 5% suggests strong robustness against that specific attack vector.

ASR is intrinsically linked to the perturbation budget (ε), which defines the maximum allowed distortion under a given Lp-norm constraint. This relationship reveals the efficiency of an attack:

• High ASR at low ε: Indicates a powerful, imperceptible attack that exploits genuine model weaknesses
• High ASR only at high ε: Suggests the attack relies on obvious, easily detectable perturbations
• L∞ norm (ε = 8/255): Standard budget for CIFAR-10 benchmarking, representing subtle pixel-level changes
• L2 norm: Measures Euclidean distance, often used for perceptual similarity constraints

Attack algorithms like Carlini & Wagner are designed to find minimal perturbations that achieve high ASR, making them more dangerous and harder to defend against than brute-force methods.

ASR varies dramatically based on the attacker's knowledge model:

• White-Box ASR: Typically approaches 100% for iterative attacks like PGD, as the attacker has full access to gradients, architecture, and parameters. This represents the worst-case vulnerability scenario.
• Black-Box ASR: Significantly lower, often 5-30% for naive query-based attacks. However, transfer-based black-box attacks can achieve 60-90% ASR by crafting adversarial examples on a surrogate model.
• Score-Based vs. Decision-Based: Score-based attacks (access to confidence scores) achieve higher ASR than decision-based attacks (only hard labels), which require more queries.

The gap between white-box and black-box ASR is a critical measure of gradient masking versus genuine robustness. A large gap often indicates a false sense of security from obfuscated gradients.

For Large Language Models, ASR measures the rate at which adversarial prompts bypass safety alignment. Key considerations include:

• Jailbreak ASR: The percentage of harmful queries that receive compliant responses after applying an adversarial suffix or prompt template
• GCG Attack Benchmarking: The Greedy Coordinate Gradient attack on Llama 2 and Vicuna models typically achieves 50-90% ASR depending on the model's safety training intensity
• Refusal Matching: Some metrics count a response as successful only if it provides genuinely harmful information, not just any non-refusal
• HarmBench Evaluation: A standardized framework that reports ASR across multiple attack types (GCG, AutoDAN, PAIR) and harm categories

ASR in LLMs must account for semantic equivalence—a jailbreak that produces a refusal with caveats may still be considered partially successful in some taxonomies.

Robust Accuracy is the complementary metric to ASR, measuring the model's classification accuracy on an adversarially perturbed test set:

• Robust Accuracy = 100% - ASR (for untargeted attacks on a perfectly clean baseline)
• A model with 90% robust accuracy under PGD-20 has an ASR of only 10%
• Adversarial Training aims to maximize robust accuracy by minimizing ASR during training
• The robustness-accuracy trade-off: Increasing robust accuracy often decreases clean accuracy, a phenomenon formalized by TRADES

Robust accuracy is the standard metric reported in academic benchmarks like RobustBench, which maintains a leaderboard of state-of-the-art defended models ranked by their robust accuracy under AutoAttack.

Rigorous ASR evaluation requires adherence to established protocols to prevent inflated claims:

• AutoAttack: A parameter-free ensemble of diverse attacks (APGD-CE, APGD-DLR, FAB, Square Attack) that provides a reliable, standardized ASR evaluation without requiring manual tuning
• RobustBench: A centralized leaderboard that enforces consistent threat models (L∞, ε=8/255) and evaluation methodologies across submissions
• Threat Model Specification: Every ASR report must declare the perturbation norm, budget (ε), number of attack iterations, and number of restarts
• Adaptive Attacks: The gold standard for defense evaluation—attacks that are specifically tailored to circumvent the defense mechanism, ensuring no gradient masking or obfuscation inflates robustness claims

Failure to use adaptive attacks has led to numerous defenses being broken shortly after publication, with ASR jumping from near-zero to near-100%.

The standard metric for measuring a model's resilience, calculated as the classification accuracy on an adversarially perturbed test set. While Attack Success Rate measures the attacker's win rate, Robust Accuracy measures the defender's. A model with 95% standard accuracy and 20% Attack Success Rate has a 75% Robust Accuracy. This metric is the primary benchmark in adversarial training research and is often reported alongside natural accuracy to quantify the accuracy-robustness trade-off.

A powerful iterative white-box attack that serves as the universal benchmark for evaluating Attack Success Rate. PGD repeatedly applies the Fast Gradient Sign Method with a small step size and projects the result back onto an epsilon-ball around the original input. It is considered a first-order adversary that provides a reliable lower bound on model robustness. If a defense claims a low Attack Success Rate against PGD, it has survived the most rigorous standard evaluation.

The process of formally proving that a model's prediction is invariant to any perturbation within a defined Lp-norm ball. Unlike empirical Attack Success Rate, which can be defeated by a stronger attack, certification provides a mathematical guarantee. Techniques include:

• Randomized Smoothing: Adds Gaussian noise and takes a majority vote
• Interval Bound Propagation: Propagates bounds through the network
• SMT Solvers: Formal verification of neural network properties A certified lower bound on accuracy is the gold standard for safety-critical systems.

The property by which an adversarial example crafted to fool one model also successfully causes misclassification in a different, independently trained model. This phenomenon dramatically increases the practical Attack Success Rate in black-box settings. An attacker can train a local substitute model, generate adversarial examples against it, and then transfer them to the target model without ever querying it. Transferability is the foundation of most practical black-box attacks.