TRADES (TRadeoff-inspired Adversarial DEfense via Surrogate-loss minimization) is an adversarial training method that optimizes a loss function balancing the trade-off between natural accuracy and robust accuracy. It achieves this by explicitly regularizing the gap between a model's prediction on a clean input and its prediction on an adversarially perturbed counterpart, using the Kullback-Leibler divergence.
Glossary
TRADES
TRADES (TRadeoff-inspired Adversarial DEfense via Surrogate-loss minimization) is an adversarial training method that optimizes a loss function balancing the trade-off between natural accuracy and robust accuracy by explicitly regularizing the gap between clean and adversarial predictions.
ADVERSARIAL DEFENSE
What is TRADES?
TRADES is a foundational adversarial training algorithm that explicitly optimizes the trade-off between a model's accuracy on clean data and its robustness to adversarial perturbations.
Unlike standard adversarial training which minimizes loss on adversarial examples alone, TRADES decomposes the objective into a natural error term and a boundary error term. This formulation provides a principled mechanism to tune the robustness-accuracy trade-off, consistently achieving state-of-the-art robustness certification results against powerful attacks like Projected Gradient Descent (PGD).
DEFENSE MECHANISM
Key Characteristics of TRADES
TRADES (TRadeoff-inspired Adversarial DEfense via Surrogate-loss minimization) is a principled adversarial training method that explicitly balances natural accuracy and robust accuracy by minimizing a boundary error.
01
The Core Objective Function
TRADES optimizes a surrogate loss that decomposes into two terms: the standard empirical risk on clean data and a regularization term that penalizes the divergence between a model's prediction on a clean input and its adversarial counterpart. This explicitly controls the trade-off between natural and robust errors, unlike standard adversarial training which conflates them.
02
Theoretical Foundation
The method is derived from a minimax formulation that upper-bounds the robust error. The key insight is that robust error can be decomposed into:
- Natural error: error on clean test data
- Boundary error: the probability that a clean example lies close to the decision boundary TRADES minimizes this boundary error term directly, pushing the decision boundary away from data points.
03
KL Divergence Regularization
The regularization term uses Kullback-Leibler (KL) divergence to measure the distance between the softmax probability distributions of clean and adversarial examples. By minimizing this divergence, TRADES enforces prediction stability—the model's output for an adversarial example must remain similar to its output for the corresponding clean example, smoothing the loss landscape.
04
Hyperparameter λ (Lambda)
A single hyperparameter λ governs the trade-off between natural accuracy and robustness:
- Higher λ: Stronger regularization, higher robust accuracy, potentially lower natural accuracy
- Lower λ: Prioritizes clean performance, reducing adversarial resilience This explicit knob provides practitioners with fine-grained control over the model's operational characteristics, unlike the implicit trade-off in PGD-based adversarial training.
05
Attack Algorithm for Perturbation Generation
To generate the adversarial example for the regularization term, TRADES uses a multi-step projected gradient descent variant that maximizes the KL divergence between the clean and perturbed predictions. This is a targeted attack on the model's output distribution, not simply a misclassification attack, ensuring the generated perturbation is maximally effective at destabilizing predictions.
06
Performance Characteristics
On benchmark datasets like CIFAR-10 under strong PGD attacks, TRADES consistently achieves superior robust accuracy compared to standard adversarial training at equivalent natural accuracy levels. It is particularly effective against white-box attacks and demonstrates strong transferability resistance, where adversarial examples crafted on one model fail to transfer to a TRADES-trained model.
METHODOLOGY COMPARISON
TRADES vs. Standard Adversarial Training
A feature-level comparison between TRADES and standard adversarial training for building robust classifiers.
| Feature | TRADES | Standard Adversarial Training |
|---|---|---|
Core Objective | Minimize natural loss + regularization term on prediction gap between clean and adversarial examples | Minimize empirical risk on adversarially perturbed training samples |
Loss Function Components | Cross-entropy on clean data + KL divergence between clean and adversarial logit distributions | Cross-entropy loss computed directly on adversarial examples |
Theoretical Foundation | Decomposes robust error into natural error and boundary error; explicitly trades off between them | Empirical defense; no explicit decomposition of error sources |
Clean Accuracy Preservation | Explicitly regularized via hyperparameter (1/λ) controlling the trade-off | Often degrades as perturbation budget increases; no explicit control mechanism |
Robustness Guarantee Type | Provable upper bound on robust error derived from PAC-Bayesian analysis | Empirical robustness only; no formal certification guarantee |
Perturbation Generation During Training | Uses PGD to find adversarial examples that maximize KL divergence from clean predictions | Uses PGD to find adversarial examples that maximize classification loss |
Hyperparameter Sensitivity | Sensitive to λ (trade-off parameter); requires tuning for dataset-specific balance | Sensitive to ε (perturbation radius) and PGD step count; no trade-off parameter |
Performance on CIFAR-10 (ε=8/255, ResNet-18) | Clean: 84.9%, Robust: 56.4% (λ=6.0) | Clean: 83.1%, Robust: 45.8% (standard PGD-AT) |
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read moreTRADES EXPLAINED
Frequently Asked Questions
Clear, technically precise answers to the most common questions about the TRADES adversarial training methodology, its mathematical foundations, and its practical implementation for building robust neural networks.
TRADES (TRadeoff-inspired Adversarial DEfense via Surrogate-loss minimization) is an adversarial training method that explicitly optimizes a loss function balancing the trade-off between natural accuracy and robust accuracy. It works by adding a regularization term to the standard empirical risk minimization objective. This term measures the Kullback-Leibler (KL) divergence between the model's prediction on a clean input and its prediction on an adversarially perturbed version of that same input. The core insight is that a robust model should not only classify correctly but also maintain consistent predictions within an epsilon-ball around each data point. The full TRADES loss function is: L(θ) = CE(f_θ(x), y) + β * KL(f_θ(x) || f_θ(x')) where x' is the adversarial example found by maximizing the KL divergence term, β is a hyperparameter controlling the trade-off, and CE is the standard cross-entropy loss for natural accuracy. This formulation directly enforces local Lipschitzness around training points, making the decision boundary smoother and less susceptible to adversarial perturbations.
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us