AutoAttack

AutoAttack removes human bias from robustness evaluation by standardizing the attack pipeline. It combines four diverse attacks without requiring manual hyperparameter tuning:

• APGD-CE: An adaptive Projected Gradient Descent variant using the cross-entropy loss with an automatic step-size scheduler.
• APGD-DLR: An adaptive PGD attack using the Difference of Logits Ratio loss, targeting models where cross-entropy gradients are masked.
• FAB-T: A targeted Fast Adaptive Boundary attack that minimizes perturbation norm against the true class.
• Square Attack: A query-efficient black-box attack that relies on random square-shaped updates, bypassing gradient-based defenses entirely.

The standard evaluation protocol operates under a strict L∞-norm threat model with a perturbation budget of 8/255 for CIFAR-10 and 4/255 for ImageNet. This formal specification ensures:

• Reproducible comparisons across different research papers and model architectures.
• A clear definition of the adversary's capabilities, preventing inflated robustness claims from weak attacks.
• Alignment with the RobustBench leaderboard, which uses AutoAttack as its primary ranking metric.

A core design goal is to expose obfuscated gradients, a brittle defense where models appear robust due to non-differentiable layers or shattered gradients. AutoAttack counters this by:

• Including the Square Attack, which does not rely on gradient information and efficiently bypasses gradient masking.
• Using adaptive step-size schedules in APGD that prevent the attack from stalling on loss plateaus.
• Providing a clear diagnostic: a large gap between white-box PGD accuracy and AutoAttack accuracy strongly indicates a flawed defense.

For defenses that claim robustness beyond the standard L∞ threat model, AutoAttack supports extended evaluation modes:

• AutoAttack+: Incorporates additional attacks like the RayS black-box method for L2-norm evaluation.
• Custom threat models: Allows specification of different perturbation budgets and norms.
• Adaptive evaluation: Researchers can integrate custom attacks tailored to specific defense mechanisms, ensuring the evaluation always matches the threat model.

Despite its ensemble nature, AutoAttack is optimized for practical evaluation:

• Early stopping: Halts attacks on samples once a successful adversarial example is found, reducing total queries.
• Batch processing: Leverages GPU parallelism for efficient gradient computation across multiple inputs.
• Fixed budget: Operates within a defined query limit, making evaluation costs predictable and comparable across hardware configurations.

The Fast Adaptive Boundary attack is a targeted attack that minimizes the Lp-norm of the perturbation required to change a model's classification. It is highly effective at finding minimal adversarial examples and is included in AutoAttack to complement the fixed-budget attacks (APGD). FAB-T specifically targets the model's decision boundary, catching vulnerabilities that loss-maximization methods might miss.

A query-efficient black-box attack that relies solely on model scores without access to gradients. It uses a randomized search scheme based on localized square-shaped updates. Its inclusion in AutoAttack ensures that defenses relying on obfuscated gradients—a common source of false robustness—are reliably broken, as the Square Attack does not depend on gradient information.

A methodology, not a specific algorithm. An adaptive attack is custom-designed to circumvent a known defense. AutoAttack is considered a strong baseline, but a truly robust model must withstand an adaptive evaluation where the attacker has full knowledge of the defense mechanism. If a defense is broken by an adaptive attack using its internal logic, the defense is considered failed, regardless of AutoAttack performance.

A brittle defense phenomenon that provides a false sense of security. It occurs when a model's gradients are non-informative (e.g., shattered, stochastic, or zero) due to non-differentiable operations or numerical instability. Gradient-based attacks like PGD fail, but black-box attacks (Square Attack) and boundary attacks (FAB-T) easily bypass it. AutoAttack's ensemble is explicitly designed to detect and defeat gradient masking.