TRADES Loss

TRADES directly addresses the fundamental tension between natural accuracy (performance on clean data) and adversarial robustness (resistance to perturbations). The loss function is formulated as:

L_TRADES = L_CE(f(x), y) + β * KL(f(x) || f(x'))

• The first term is standard cross-entropy loss for clean accuracy
• The second term is the Kullback-Leibler divergence between the clean output distribution and the adversarial output distribution
• The hyperparameter β controls the trade-off: higher β prioritizes robustness over accuracy

Unlike adversarial training which uses the label of the adversarial example directly, TRADES uses the model's own prediction on the clean input as the soft target for the adversarial example. This encourages output stability:

• The model learns to produce similar probability distributions for clean and perturbed inputs
• This prevents the model from becoming overconfident on adversarial examples
• The KL divergence term acts as a local smoothness regularizer around each data point
• Results in smoother decision boundaries that are harder to exploit

TRADES generates adversarial examples using Projected Gradient Descent (PGD) to maximize the KL divergence between clean and perturbed outputs:

• The perturbation is constrained within an Lp-norm ball (typically L∞ with radius ε)
• Uses multi-step PGD with random initialization for stronger attacks
• The inner maximization finds the perturbation that causes the largest output distribution shift
• This is more aligned with robustness goals than maximizing classification error alone

TRADES provides a principled upper bound on the robust error of a classifier. The theoretical decomposition shows:

• Robust error ≤ Natural error + Boundary error
• Natural error is the error on clean samples
• Boundary error measures how often the model's prediction changes within the perturbation neighborhood
• TRADES minimizes both terms simultaneously
• This formalization explains why robustness and accuracy can conflict and provides a framework for managing the trade-off

Standard adversarial training (Madry et al.) uses one-hot labels for adversarial examples, while TRADES uses the model's own soft predictions on clean data:

• Standard AT: L_CE(f(x'), y) — forces correct classification of perturbed inputs
• TRADES: L_CE(f(x), y) + β * KL(f(x) || f(x')) — forces output consistency
• TRADES often achieves higher clean accuracy at equivalent robustness levels
• Particularly effective when β is tuned appropriately for the dataset and threat model
• Widely adopted as a strong baseline in adversarial robustness research

Key implementation details for effective TRADES training:

• β selection: Typically ranges from 1.0 to 6.0; higher values for stronger robustness requirements
• PGD steps: 10-step PGD is standard during training; more steps increase computational cost
• Step size: Usually set to ε/4 for L∞ perturbations with radius ε = 8/255
• Training time: Approximately 2-3x slower than standard training due to adversarial example generation
• Evaluation: Use AutoAttack for reliable robustness assessment rather than PGD alone
• Available in libraries like Adversarial Robustness Toolbox (ART) and RobustBench

The foundational defensive technique that augments training data with adversarial examples. TRADES is a specific adversarial training algorithm that improves upon standard approaches by explicitly optimizing the trade-off between natural accuracy and robustness.

• Standard AT minimizes loss on adversarial examples only
• TRADES minimizes a clean loss plus a regularization term
• The regularization term is the KL divergence between clean and adversarial output distributions

The standard white-box attack used to generate adversarial examples during TRADES training. PGD iteratively perturbs an input within an Lp-norm ball to maximize the loss.

• TRADES uses PGD to generate the adversarial variant of each clean input
• The perturbation budget epsilon defines the maximum allowed distortion
• Multi-step PGD with random starts is the de facto evaluation standard

The statistical distance measure at the heart of TRADES. It quantifies how much one probability distribution diverges from a reference distribution.

• TRADES minimizes KL(p_clean || p_adversarial)
• This forces the model's output on adversarial inputs to match its output on clean inputs
• Unlike cross-entropy, KL divergence is asymmetric and captures distributional shift