Ensemble Adversarial Training

Unlike standard adversarial training which uses a single model, EAT generates perturbations from a diverse collection of held-out or surrogate models. This prevents the defender from overfitting to a specific attack vector.

• Surrogate Pool: Includes models with varied architectures (ResNets, DenseNets, ViTs) and training seeds.
• Transferability: Adversarial examples that transfer across multiple models are more likely to lie in the true blind-spots of the data manifold.
• Black-Box Hardening: The resulting model is robust even when the attacker has no access to the defender's exact gradients.

EAT often augments data not just with hard-label attacks but with soft-label perturbations. By matching the output distributions of diverse teacher models, the student model learns smoother decision boundaries.

• Soft Targets: Minimizes KL divergence between the defender's output and the ensemble's average prediction.
• Gradient Regularization: The variance in gradients across the ensemble acts as a natural regularizer, penalizing sharp curvature in the loss landscape.
• Knowledge Distillation: This process effectively distills the robust features of multiple teachers into a single compact student model.

A critical failure mode in single-model adversarial training is gradient masking, where the model learns to obfuscate its gradients rather than truly resist perturbations. EAT naturally breaks this cycle.

• Diverse Loss Landscapes: Since surrogates have distinct loss surfaces, the attacker cannot easily find a single masking pattern that fools all models.
• Stochastic Defense: The random selection of surrogate models during training introduces an information asymmetry that thwarts deterministic gradient-based attackers.
• Reliable Robustness: EAT provides a more honest robustness signal, avoiding the false sense of security often seen with obfuscated gradients.

The primary trade-off of EAT is the significant increase in computational cost. Generating attacks for N models requires N forward and backward passes per training step.

• Parallelization: Surrogate attacks can be distributed across multiple accelerators to maintain throughput.
• Subset Sampling: Practical implementations often sample a random subset of surrogates per batch rather than using the full ensemble.
• Frozen Surrogates: Pre-computing adversarial examples with frozen surrogate models offline can decouple attack generation from the main training loop, drastically reducing wall-clock time.

EAT is a natural complement to other state-of-the-art robustness frameworks. It specifically enhances the boundary error term in the TRADES loss function.

• TRADES Integration: The TRADES loss trades off natural accuracy and robustness; EAT provides a superior estimate of the boundary error by using multi-model perturbations.
• Pseudo-Labeling: In semi-supervised settings, an ensemble of teachers generates more reliable pseudo-labels for unlabeled data, improving robust self-training loops.
• Certified Radius: While EAT is an empirical defense, combining it with randomized smoothing can provide probabilistic certificates against Lp-norm attacks.

EAT is particularly effective against physical-world attacks where the attacker must craft a perturbation that transfers across unknown environmental conditions and sensor variations.

• Expectation over Transformation (EoT): EAT naturally extends EoT by averaging attacks over both model parameters and input transformations.
• Adversarial Patches: Ensembles trained with varied patch shapes and positions are significantly more robust to localized physical attacks.
• Sensor Diversity: In autonomous systems, an ensemble can represent different camera sensors, hardening the perception stack against universal physical perturbations.