Robust Self-Training

The process begins with a teacher model generating hard or soft pseudo-labels for a large pool of unlabeled data. A student model is then trained on this augmented dataset. In robust self-training, the teacher is often an adversarially pre-trained model or an ensemble, ensuring the pseudo-labels are not brittle. The student model is trained to match these labels even under adversarial perturbation, preventing the amplification of the teacher's errors. This contrasts with standard self-training, where the student merely mimics the teacher on clean data, often inheriting its vulnerabilities.

Unlike standard self-training which minimizes standard cross-entropy on pseudo-labels, robust self-training injects an adversarial loss term. The student model is optimized using a min-max formulation: it minimizes the loss against a worst-case adversary that maximizes the loss within an epsilon-ball around the unlabeled input. This forces the decision boundary to lie in low-density regions of the data manifold. Common implementations use Projected Gradient Descent (PGD) to craft adversarial examples on-the-fly from unlabeled data, ensuring the model learns representations invariant to small perturbations.

To prevent the model from collapsing due to noisy pseudo-labels, robust self-training employs strict confidence thresholding. Only unlabeled examples where the teacher model assigns a maximum class probability above a high threshold (e.g., 0.95) are retained. This is often combined with consistency regularization, where the student's prediction on a clean unlabeled image must match its prediction on an adversarially perturbed version of the same image. This dual constraint ensures that the model does not learn to confidently misclassify adversarial examples.

Instead of hard one-hot pseudo-labels, robust self-training often uses soft pseudo-labels (probability distributions over classes). This transfers the teacher's uncertainty and class-relationship knowledge to the student, acting as a form of robust knowledge distillation. The student is trained to minimize the Kullback-Leibler divergence between its own output distribution and the teacher's soft labels, both on clean and adversarially perturbed inputs. This prevents the student from overfitting to the teacher's incorrect hard decisions and smooths the loss landscape.

Advanced robust self-training pipelines leverage generative models to augment the unlabeled dataset. Instead of relying solely on standard geometric augmentations, a Variational Autoencoder (VAE) or Generative Adversarial Network (GAN) is used to generate realistic, challenging samples near the decision boundary. The teacher model labels these generated samples, and the student is forced to classify them consistently. This exposes the student to a wider variety of semantic-preserving perturbations than standard Lp-norm adversarial attacks alone.

Robust self-training is not a single-pass operation. It operates in iterative rounds. After the student model converges, it becomes the new teacher for the next round, re-labeling the unlabeled data pool. Crucially, the adversarial robustness budget is maintained or increased in each round. This iterative process gradually refines the decision boundary and can correct initial labeling mistakes made by a weaker teacher. The process halts when the adversarial accuracy on a validation set plateaus, preventing overfitting to the pseudo-labeling objective.

A foundational defensive technique that augments training datasets with adversarial examples—inputs intentionally perturbed to cause misclassification. By training on these malicious samples alongside clean data, the model learns to maintain accurate predictions even under attack.

• Mechanism: Injects AEs into each training batch
• Goal: Flatten the loss landscape around data points
• Synergy with RST: Pseudo-labeled data can be adversarially perturbed before student training

An algorithm that optimizes the trade-off between natural accuracy and adversarial robustness by decomposing prediction error into natural and boundary errors. It uses a regularization term based on the Kullback-Leibler divergence between clean and adversarial output distributions.

• Key insight: Robustness need not catastrophically sacrifice clean accuracy
• Relevance: Provides the theoretical loss function often adapted for robust self-training objectives

A semi-supervised regularization method that smooths the model's output distribution by minimizing the KL divergence between predictions on clean unlabeled data and locally perturbed versions of that same data.

• No labels required: Operates purely on input consistency
• Perturbation direction: Computed to maximize output distribution change
• RST connection: VAT is a precursor concept—RST extends this with pseudo-labeling and stronger adversarial objectives

A powerful iterative white-box attack that generates adversarial examples by taking multiple small gradient steps and projecting the perturbation back onto an epsilon-ball constraint after each iteration.

• Standard steps: 40 iterations with step size ε/10
• Role in RST: Often used as the inner-loop adversary to generate the worst-case perturbations for robust pseudo-labeling

A false sense of security phenomenon where a defense produces obfuscated or shattered gradients, preventing gradient-based attacks from succeeding while leaving the model vulnerable to black-box or transfer attacks.

• Causes: Non-differentiable layers, numerical instability, saturating activations
• Detection: Verify defense with black-box attacks and transferability tests
• RST caution: Robust self-training must avoid inadvertently inducing gradient masking through its teacher-student dynamics

The mathematical framework underpinning adversarial robustness, formulated as minimizing the empirical risk against a worst-case adversary that maximizes the loss within a perturbation set.

• Inner maximization: Attacker finds strongest perturbation
• Outer minimization: Defender adjusts weights to resist that perturbation
• RST formulation: Extends this to semi-supervised settings where the inner loop also considers consistency with pseudo-labels on unlabeled data