RobustBench

Enforces a strict, common evaluation protocol based on a specific threat model to prevent methodological inconsistencies. This includes a fixed L-infinity perturbation budget (typically ε=8/255) and a defined set of allowed attacker knowledge. By standardizing these parameters, RobustBench eliminates the common pitfall of comparing defenses evaluated under different, often weaker, attack settings, ensuring a level playing field for all submissions.

Uses AutoAttack (AA) as the default, parameter-free benchmark for evaluation. AutoAttack is an ensemble of four diverse attacks:

• APGD-CE: An adaptive Projected Gradient Descent attack with a cross-entropy loss.
• APGD-DLR: An adaptive PGD attack using the Difference of Logits Ratio loss.
• FAB-T: A targeted version of the Fast Adaptive Boundary attack.
• Square Attack: A query-efficient black-box attack. This ensemble provides a reliable and strong empirical robustness estimate, replacing weaker, obsolete attacks.

Maintains a dynamic, publicly accessible leaderboard that ranks defenses by their robust accuracy against AutoAttack on standard datasets like CIFAR-10, CIFAR-100, and ImageNet. The leaderboard tracks both clean accuracy and adversarial accuracy, explicitly highlighting the trade-off between standard performance and robustness. This transparency allows researchers to quickly identify the state-of-the-art and prevents inflated claims of robustness from non-adaptive evaluations.

Provides a curated Model Zoo containing pre-trained, robust models from top-ranked defenses. These models are readily accessible for download and use, serving as standardized baselines for new research. The zoo includes models with varying levels of robustness, enabling researchers to:

• Benchmark new attacks against known defenses.
• Perform transfer attack studies.
• Use robust models as feature extractors for downstream tasks without retraining from scratch.

Implements a strict submission process that requires defenders to provide a full, executable code package. Each submission is independently re-evaluated by the RobustBench maintainers using the standard AutoAttack protocol. This process is designed to detect and reject defenses that rely on gradient masking or other obfuscation techniques that create a false sense of security. The protocol mandates that a defense must be effective against a full adaptive attack to be considered valid.

A parameter-free ensemble of diverse white-box and black-box attacks that serves as the default evaluation protocol on RobustBench. AutoAttack combines APGD-CE, APGD-DLR, FAB, and Square Attack to provide a reliable, adaptive assessment that resists gradient masking and other obfuscated defenses. Its deterministic, tuning-free nature eliminates human bias in robustness claims.

The dominant defense paradigm on RobustBench, where models are trained on adversarial examples generated on-the-fly. The most effective variants use TRADES loss or AWP (Adversarial Weight Perturbation) to balance natural accuracy with robust generalization. This technique directly minimizes the model's worst-case loss within a defined perturbation budget, typically an L-infinity ball of radius 8/255.

A probabilistic certification method that constructs a smoothed classifier by adding isotropic Gaussian noise to inputs. Unlike empirical defenses, it provides a provable radius within which no adversarial example can exist. RobustBench tracks certified accuracy separately, as the guarantees come at a significant cost to natural accuracy compared to empirical methods.

RobustBench standardizes evaluation around specific perturbation constraints. The most common is the L-infinity norm (pixel-wise maximum change), followed by L2 (Euclidean distance). Each threat model defines the adversary's power. A defense validated only on L-infinity may fail catastrophically against L1, spatial transformations, or common corruptions, making multi-norm evaluation critical.

An evaluation methodology where the attacker has full knowledge of the defense and tailors the attack to circumvent it. RobustBench enforces this by requiring defenses to be open-source and reproducible. If a defense relies on gradient masking, randomized transformations, or non-differentiable components, an adaptive attack will expose the false sense of security by using BPDA (Backward Pass Differentiable Approximation) or EOT (Expectation over Transformation).