RobustBench is a curated evaluation framework that standardizes the comparison of adversarial robustness across different neural network architectures. It mitigates the common pitfall of gradient masking by providing a fixed set of pre-evaluated model checkpoints and strictly defined threat models, ensuring that reported gains in robustness are genuine and not artifacts of flawed evaluation methodologies.
Glossary
RobustBench
A standardized benchmark and leaderboard for adversarial robustness that provides a curated set of model checkpoints and attack evaluations to ensure reproducible and comparable results.
ADVERSARIAL ROBUSTNESS STANDARDIZATION
What is RobustBench?
RobustBench is a standardized benchmark and public leaderboard designed to provide a rigorous, reproducible evaluation of adversarial robustness in image classification models.
The leaderboard relies on AutoAttack, a parameter-free ensemble of diverse white-box and black-box attacks, to provide a reliable and consistent measurement of empirical robustness. By enforcing a common evaluation protocol, RobustBench has become the definitive reference for tracking progress in adversarial training techniques like TRADES and for establishing state-of-the-art certified robustness claims.
STANDARDIZED ADVERSARIAL ROBUSTNESS
Key Features of RobustBench
RobustBench provides a curated, standardized leaderboard and model zoo for adversarial robustness, enabling reproducible and comparable evaluation of defenses against strong attacks.
01
Standardized Threat Model
Enforces a strict threat model based on an L-infinity norm perturbation budget (typically epsilon = 8/255). This prevents defenses from exploiting obfuscated gradients or unrealistic assumptions. Every model is evaluated under the same white-box access conditions, ensuring that comparisons are fair and that reported robustness is not an artifact of gradient masking.
02
AutoAttack Evaluation Suite
Uses AutoAttack (AA) as the primary evaluation metric, a parameter-free ensemble of diverse attacks including:
- APGD-CE: Auto-PGD on cross-entropy loss
- APGD-DLR: Auto-PGD on difference of logits ratio
- FAB-T: Fast Adaptive Boundary attack
- Square Attack: Query-efficient black-box attack This ensemble prevents defenses from overfitting to a single attack method and provides a reliable measure of empirical robustness.
03
Curated Model Zoo
Provides a repository of over 100 pre-trained model checkpoints with verified robustness claims. Each model is stored with its exact architecture, training recipe, and AutoAttack evaluation results. This eliminates the common problem of researchers reporting results on different evaluation setups, enabling direct, apples-to-apples comparisons without re-implementing complex training procedures.
04
Leaderboard Transparency
Maintains a public leaderboard ranking models by robust accuracy on CIFAR-10, CIFAR-100, and ImageNet. Each entry includes:
- Standard accuracy on clean data
- Robust accuracy under AutoAttack
- Model architecture and parameter count
- Link to the original paper and checkpoint This transparency exposes the accuracy-robustness trade-off and drives the community toward genuinely robust architectures.
05
Reproducibility Guarantee
All models on the leaderboard are independently re-evaluated by the RobustBench maintainers using a fixed evaluation protocol. This eliminates the common pitfall of self-reported robustness where subtle implementation bugs or weak attack hyperparameters inflate robustness claims. The fixed evaluation codebase is open-source, allowing anyone to verify results.
06
Integration with Adversarial Training Research
Serves as the de facto benchmark for evaluating new adversarial training methods. Techniques like TRADES, MART, and AWP (Adversarial Weight Perturbation) are all ranked on the leaderboard. Researchers can download baseline models, run their defense, and submit results for independent verification, accelerating the pace of reproducible robustness research.
ROBUSTBENCH EXPLAINED
Frequently Asked Questions
Clear, technically precise answers to the most common questions about the RobustBench leaderboard, its evaluation methodology, and its role in standardizing adversarial robustness research.
RobustBench is a standardized benchmark and public leaderboard for evaluating the adversarial robustness of image classification models. It works by providing a curated, fixed set of pre-trained model checkpoints and a canonical evaluation protocol based on AutoAttack, a parameter-free ensemble of attacks. Rather than relying on authors to self-report robustness, RobustBench applies a consistent, strong attack suite to every submitted model, ensuring that reported accuracy under attack is directly comparable and reproducible. The framework eliminates common pitfalls like gradient masking and weak attack evaluations, establishing a trusted, apples-to-apples comparison for the research community.
ADVERSARIAL ROBUSTNESS EVALUATION COMPARISON
RobustBench vs. Other Evaluation Methods
A comparison of RobustBench against traditional evaluation approaches for measuring adversarial robustness, highlighting standardization, reproducibility, and threat model coverage.
| Feature | RobustBench | Ad-hoc Evaluation | AutoAttack Standalone |
|---|---|---|---|
Standardized threat model | |||
Curated model zoo with checkpoints | |||
Reproducible leaderboard rankings | |||
Parameter-free attack ensemble | |||
Defense-adaptive attack selection | |||
Gradient masking detection | |||
L-infinity epsilon = 8/255 default | |||
Standard accuracy vs. robustness trade-off tracking |
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read more
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us