Adversarial Training

Adversarial training is formally defined as a min-max optimization problem. The inner maximization step generates the strongest possible adversarial examples that maximize the model's loss, while the outer minimization step updates model parameters to correctly classify those perturbed samples. This saddle-point formulation, introduced by Madry et al. (2018), is the theoretical foundation of modern adversarial robustness. The process effectively trains the model on worst-case inputs rather than just clean data.

• Inner loop: Projects perturbations within an Lp-norm ball (typically L∞ with radius ε=8/255 for images)
• Outer loop: Standard empirical risk minimization on the generated adversarial examples
• Key insight: The model learns to be locally stable around each training point

The most effective adversarial training methods use multi-step iterative attacks to generate training examples. PGD starts from a random initialization within the epsilon-ball and takes multiple gradient steps, projecting back onto the norm constraint after each step. Variants include PGD with random restarts, CW (Carlini-Wagner) attacks optimized for minimal distortion, and AutoAttack ensembles. The number of attack iterations directly correlates with robustness—single-step methods like FGSM are insufficient and lead to catastrophic overfitting.

• PGD-k: k-step variant; k=10 is common, k=40 is rigorous
• Random start: Critical to avoid gradient masking
• AutoAttack: Parameter-free ensemble for standardized evaluation

TRADES (TRadeoff-inspired Adversarial DEfense via Surrogate-loss) explicitly balances natural accuracy against adversarial robustness through a regularized loss function. Instead of using only adversarial examples for training, TRADES minimizes the KL divergence between clean and adversarial output distributions while maintaining standard cross-entropy on clean data. The hyperparameter 1/λ controls the trade-off: higher values prioritize robustness over clean accuracy. This formulation addresses the fundamental tension where improving robustness often degrades performance on unperturbed inputs.

• Loss = CE(clean) + β · KL(clean || adversarial)
• β controls robustness-accuracy balance
• Empirically superior to vanilla adversarial training on multiple benchmarks

Adversarial training imposes a significant computational burden. Each training iteration requires generating adversarial examples, which involves multiple forward and backward passes through the network. For a PGD-10 attack, this multiplies computation by roughly 11x (10 attack steps + 1 parameter update). Techniques to mitigate this include free adversarial training (replaying gradient computations), fast adversarial training with single-step attacks plus regularization, and mixed-precision training. Despite the cost, adversarial training remains the only empirically robust defense against strong adaptive attacks.

• PGD-10: ~11x compute vs. standard training
• Free AT (Shafahi et al.): Reuses gradients across minibatches
• Fast AT (Wong et al.): FGSM + random initialization + early stopping

Static adversarial training with a fixed perturbation budget can be suboptimal. Curriculum adversarial training progressively increases the attack strength (epsilon radius or PGD iterations) during training, allowing the model to first learn robust features at smaller radii before facing stronger attacks. Dynamic scheduling adjusts the perturbation budget based on the model's current robustness, ensuring the adversary remains challenging but not overwhelming. This approach improves convergence stability and final robustness compared to training with a fixed, maximum-strength adversary from the start.

• Epsilon scheduling: Start small, increase over epochs
• Adaptive attacks: Match perturbation strength to model capacity
• Convergence benefit: Avoids early training instability from overly strong adversaries

In financial fraud detection, adversarial training must account for feature-space constraints unique to tabular data. Unlike image pixels, transaction features have heterogeneous types (categorical, continuous, ordinal) and semantic constraints (e.g., transaction amount cannot be negative). Constrained adversarial training generates perturbations that respect these domain rules. Additionally, adversaries in finance operate under asymmetric cost structures—a false negative (missed fraud) is far more costly than a false positive. Adversarial training can be weighted to prioritize robustness on high-risk transaction segments.

• Tabular constraints: Respect feature semantics and business rules
• Cost-sensitive weighting: Prioritize recall on high-value transactions
• Feature grouping: Apply different perturbation budgets per feature type