AutoAttack

AutoAttack eliminates the need for manual hyperparameter tuning by combining four diverse attacks: two white-box attacks (APGD-CE and APGD-DLR) and two black-box attacks (FAB and Square Attack). This ensemble automatically adapts to the defense, preventing gradient masking from fooling the evaluation. The parameter-free nature ensures reproducible and comparable results across different models and defenses, making it the standard for RobustBench leaderboard evaluations.

The core white-box component uses Auto-PGD, an improved Projected Gradient Descent that automatically adjusts the step size during the attack. It comes in two loss variants:

• APGD-CE: Uses cross-entropy loss, effective against standard models
• APGD-DLR: Uses the Difference of Logits Ratio loss, specifically designed to bypass defenses that rely on logit masking or label smoothing Both variants incorporate a momentum term and restart mechanism to escape local optima.

The Fast Adaptive Boundary (FAB) attack is a white-box method that minimizes the Lp norm of perturbations by iteratively projecting adversarial examples onto the decision boundary. In AutoAttack, FAB is used to generate minimally perturbed adversarial examples, complementing the fixed-epsilon APGD attacks. This combination ensures that defenses relying on obfuscated gradients or non-smooth loss surfaces are reliably evaluated and not incorrectly labeled as robust.

The Square Attack is a query-efficient, score-based black-box attack that requires only the model's predicted class probabilities. It iteratively selects random square-shaped perturbations and accepts updates that improve the attack objective. Unlike gradient-based methods, Square Attack is immune to gradient masking defenses, making it essential for detecting false robustness claims. Its inclusion ensures AutoAttack evaluates true empirical robustness, not just gradient obfuscation.

AutoAttack is the default evaluation protocol for RobustBench, the leading benchmark for adversarial robustness. It provides:

• Reproducible evaluations with fixed attack parameters
• Threat model specification using L-infinity or L2 norm constraints
• Reliable ranking of defenses by measuring robust accuracy under the ensemble This standardization has exposed numerous defenses that relied on gradient masking rather than true robustness, advancing the field toward verifiable security claims.

AutoAttack operates under a defined threat model with specific perturbation constraints:

• L-infinity norm: Bounded pixel-wise perturbations (e.g., epsilon = 8/255)
• L2 norm: Bounded Euclidean distance perturbations
• Standard version: Combines APGD-CE, APGD-DLR, FAB, and Square Attack
• Rand version: Adds random restarts for stronger evaluation The choice of norm and epsilon defines the adversarial budget, and AutoAttack provides a reliable lower bound on model robustness within these constraints.

The iterative white-box attack that forms the backbone of AutoAttack's APGD component. PGD takes multiple gradient steps, each time projecting the perturbed input back onto an epsilon-ball around the original sample.

• Key mechanism: Iterative FGSM with projection
• Standard steps: 20–100 iterations
• Norm constraint: Typically L-infinity
• Role in AutoAttack: APGD-CE and APGD-DLR are momentum-accelerated variants

The primary defensive methodology against gradient-based attacks like those in AutoAttack. During training, adversarial examples are generated on-the-fly and added to the minibatch, forcing the model to learn a robust decision boundary.

• Standard approach: PGD-based adversarial training
• Trade-off: Clean accuracy often decreases
• Evaluation: AutoAttack is the de facto standard for measuring success
• Key insight: Defenses not evaluated with AutoAttack often exhibit gradient masking

A brittle defense phenomenon where a model's gradients are intentionally or unintentionally obfuscated, causing gradient-based attacks to fail. AutoAttack's ensemble approach explicitly counters this by including black-box attacks like Square Attack.

• Symptoms: Shattered gradients, stochastic gradients, vanishing gradients
• Detection: Black-box attacks succeed where white-box fail
• AutoAttack's role: Exposes masked gradients via Square Attack and FAB
• Key lesson: High PGD accuracy without AutoAttack validation is meaningless

Unlike AutoAttack's empirical evaluation, certified robustness provides a mathematical guarantee that no perturbation within a specified bound can change the prediction. Randomized smoothing is the most common technique.

• Empirical vs. Certified: AutoAttack gives upper bounds; certification gives lower bounds
• Gap: Certified accuracy is typically much lower than empirical robust accuracy
• Complementary use: AutoAttack for practical assessment; certification for safety-critical guarantees
• Key metric: Certified radius under L2 or L-infinity norm

The formal characterization of an adversary's capabilities that defines the scope of AutoAttack's evaluation. Specifying the threat model is essential for meaningful robustness claims.

• Knowledge: White-box (full gradient access) or black-box (query access only)
• Perturbation norm: L-infinity (most common), L2, L0
• Epsilon budget: Maximum allowed perturbation magnitude (e.g., 8/255 for CIFAR-10)
• AutoAttack default: L-infinity, epsilon = 8/255, full model access