TRADES

TRADES decomposes the learning problem into two components: empirical risk minimization on natural examples and a robustness regularization term. The objective function is:

min_f E[L(f(x), y) + λ · max_{x'∈B(x,ε)} D(f(x), f(x'))]

• L(f(x), y): Standard classification loss on clean data
• D(f(x), f(x')): Divergence between predictions on clean and adversarial examples (typically Kullback-Leibler divergence)
• λ: Tunable hyperparameter controlling the accuracy-robustness trade-off
• B(x,ε): Allowed perturbation ball around input x

This formulation directly optimizes the theoretical upper bound on adversarial risk derived by Zhang et al. (2019).

Unlike standard adversarial training which can overfit to a specific attack strength, TRADES provides explicit control over the trade-off curve:

• Small λ (e.g., 1.0): Prioritizes natural accuracy, suitable when clean performance is critical
• Large λ (e.g., 6.0): Prioritizes adversarial robustness, ideal for security-critical applications
• KL Divergence Regularization: Penalizes output instability rather than forcing correct classification on adversarial examples, leading to smoother decision boundaries

The framework provably bounds the gap between natural and robust error, giving practitioners a principled knob rather than an opaque heuristic.

TRADES employs Projected Gradient Descent (PGD) to approximate the worst-case perturbation during training:

• Inner Loop: Iteratively maximizes the divergence D(f(x), f(x')) within the ε-ball
• Step Size: Typically set to ε/4 with 10 iterations for CIFAR-10 scale problems
• Projection: After each gradient step, the perturbed input is projected back onto the L∞ ball of radius ε around the clean input
• Gradient Propagation: The outer minimization backpropagates through the entire inner maximization process

This bi-level optimization is computationally intensive but yields significantly more robust models than single-step FGSM-based defenses.

TRADES generalizes and improves upon Madry et al.'s adversarial training formulation:

• Standard AT: Minimizes loss on PGD-generated adversarial examples directly — min E[max L(f(x'), y)]
• TRADES: Separates natural accuracy from robustness — optimizes clean loss plus prediction stability
• Key Advantage: TRADES does not require adversarial examples to be correctly labeled; it only requires the model's output distribution to remain stable under perturbation
• Empirical Results: On CIFAR-10 under L∞ ε=8/255, TRADES achieves ~56% robust accuracy vs ~47% for standard AT at comparable natural accuracy levels

This decoupling makes TRADES particularly effective against strong iterative attacks like AutoAttack.

The TRADES objective has deep theoretical connections to certified robustness and randomized smoothing:

• Upper Bound Minimization: TRADES directly minimizes a differentiable surrogate of the robust error upper bound
• Smoothness Induction: The KL divergence penalty encourages locally Lipschitz-continuous models, a prerequisite for certification
• Synergy with Randomized Smoothing: Models trained with TRADES serve as stronger base classifiers for randomized smoothing, yielding tighter certified radii
• Provable Guarantees: Under certain conditions, the TRADES loss provides a certified robustness guarantee against Lp-bounded perturbations

This theoretical grounding distinguishes TRADES from purely empirical defense methods.

Practical deployment of TRADES requires careful engineering:

• Batch Normalization: Use separate batch statistics for clean and adversarial forward passes to avoid distribution mismatch
• Mixed Precision Training: The bi-level optimization benefits from FP16 acceleration but requires gradient scaling
• Early Stopping: Monitor both clean and robust validation accuracy; robust accuracy often plateaus before clean accuracy
• Multi-GPU Distribution: The inner PGD loop is embarrassingly parallel across batch dimensions
• Learning Rate Scheduling: Cosine annealing with warm restarts typically outperforms step decay for TRADES

Reference implementation available in the Adversarial Robustness Toolbox (ART) and official TRADES GitHub repository.