TRADES

TRADES mathematically separates a classifier's prediction error into two distinct sources: natural error (performance on clean data) and boundary error (performance near decision boundaries). The algorithm then optimizes a surrogate loss that minimizes the sum of these errors, creating a tunable trade-off parameter (λ) that controls the balance between standard accuracy and adversarial resilience. This decomposition provides theoretical grounding for why robust models often sacrifice clean accuracy.

The core loss function is: min_f E[L(f(x), y) + λ · max_{x'∈B(x,ε)} L(f(x), f(x'))]. The first term enforces natural accuracy by minimizing cross-entropy on clean inputs. The second term enforces robustness by minimizing the KL divergence between predictions on clean data and their worst-case adversarial perturbations within an ε-ball. The hyperparameter λ directly controls the robustness-accuracy trade-off, with higher values prioritizing adversarial resilience.

Unlike single-step methods like FGSM, TRADES employs Projected Gradient Descent (PGD) with multiple iterations (typically 10-20 steps) to craft the inner maximization's adversarial examples. Each step takes a gradient ascent move toward maximizing the KL divergence between clean and perturbed predictions, then projects the perturbation back onto the L∞ ε-ball constraint. This iterative approach generates stronger adversaries than single-step methods, leading to more robust trained models.

TRADES provides provable upper bounds on the robust error of the learned classifier. The algorithm is derived from a theoretical analysis showing that the robust error is bounded by the sum of natural error and boundary error. By minimizing this bound directly, TRADES achieves state-of-the-art certified robustness under the L∞ threat model. This theoretical foundation distinguishes TRADES from heuristic adversarial training approaches that lack formal guarantees.

Standard adversarial training (Madry et al.) uses adversarial labels—generating perturbed examples and training with their original labels. TRADES instead uses clean labels for the natural loss and encourages prediction consistency between clean and adversarial views via KL divergence. This key difference means TRADES doesn't force the model to classify perturbed inputs identically to clean ones, but rather to maintain similar output distributions, resulting in smoother decision boundaries and often better clean accuracy at equivalent robustness levels.

Key implementation considerations include: λ selection (commonly 1.0-6.0, with 6.0 used in the original paper for CIFAR-10), step size for PGD (typically ε/4 for L∞ attacks), and number of PGD steps (10 during training, 20+ for evaluation). TRADES is often combined with early stopping and learning rate scheduling (e.g., piecewise decay at epochs 75 and 90 for 100-epoch training). The algorithm is implemented in the Adversarial Robustness Toolbox (ART) and widely available in open-source repositories.