Gradient Masking

The loss surface is intentionally made non-differentiable or numerically unstable to prevent gradient computation.

• Mechanism: Introducing non-differentiable operations like hard thresholding, quantization, or thermometer encoding into the model's forward pass
• Effect: Gradient-based optimizers receive zero or meaningless gradient signals, halting attack convergence
• Bypass: Attackers substitute the non-differentiable component with a smooth approximation during the backward pass (Backward Pass Differentiable Approximation)
• Example: Defensive distillation at high temperatures creates near-zero gradients that collapse numerical precision

The model or defense introduces randomness into the inference process, causing gradients to become noisy and unreliable for optimization.

• Mechanism: Random resizing, random padding, random noise injection, or stochastic activation functions applied to inputs or intermediate layers
• Effect: Each gradient computation yields different results, preventing attackers from finding a consistent descent direction
• Bypass: Apply Expectation Over Transformation (EOT) to compute the average gradient over the randomization distribution
• Critical flaw: Randomness alone does not remove the adversarial direction; it only hides it in noise that can be averaged out

The defense architecture causes gradients to saturate or explode during backpropagation, making optimization numerically impossible.

• Mechanism: Deep computational graphs with saturating activation functions (sigmoid, tanh) or repeated evaluation loops that amplify gradient magnitudes
• Effect: Gradient values approach zero (vanishing) in flat saturation regions or overflow to infinity (exploding), preventing meaningful parameter updates
• Bypass: Use Carlini & Wagner (C&W) attack with a margin-based loss function that avoids saturation by operating directly on logits rather than softmax probabilities
• Indicator: A model that appears perfectly robust (0% attack success rate) under basic PGD but collapses under C&W optimization

A unifying term describing any defense where gradients exist mathematically but are misleading or non-informative for attack optimization.

• Identified by Athalye et al. (2018): A landmark paper that systematically broke 7 of 9 ICLR 2018 defenses by demonstrating they relied on gradient obfuscation
• Detection tests:
  • One-step attack superiority: Single-step FGSM outperforms iterative PGD, indicating gradient information degrades with more steps
  • Increasing distortion bound: Attack success does not increase with larger epsilon, suggesting the attack is stuck in a flat region
  • Black-box transfer: Black-box attacks succeed where white-box attacks fail, confirming the defense only masks gradients
• Key insight: True adversarial robustness requires gradient transparency, not gradient concealment

A training procedure where a teacher model's soft probability outputs are used to train a student model, originally proposed as an adversarial defense.

• Mechanism: Train a model at high temperature, use its soft labels (class probabilities) to train a second model at the same temperature, then deploy at temperature 1
• Claimed effect: Smooths the decision surface and reduces gradient magnitude, making it hard for attackers to find adversarial directions
• Reality: Papernot et al. (2016) initially claimed robustness, but Carlini & Wagner (2016) demonstrated that distillation merely masks gradients through saturation
• Bypass: C&W attack operates on logit-space (pre-softmax) values, completely circumventing the temperature-induced gradient obfuscation
• Lesson: Distillation is now understood as a classic case study in gradient masking, not a viable standalone defense

The only reliable methodology for determining whether a defense provides true robustness or merely gradient masking.

• Principle: Assume the attacker has full knowledge of the defense mechanism and can modify their attack to specifically circumvent it
• Required steps:
  • Identify any non-differentiable components and replace them with differentiable approximations
  • Apply EOT over any stochastic transformations
  • Use loss functions that avoid gradient saturation (logit-space optimization)
  • Verify that increasing perturbation budget increases attack success
• Red flag: If a defense paper only evaluates against standard PGD without adaptive modifications, the claimed robustness is likely gradient masking
• Standard practice: Security reviewers now require adaptive attack evaluation before accepting any new defense proposal

The primary empirical defense against adversarial examples. This technique augments the training dataset with Projected Gradient Descent (PGD) attacks generated on-the-fly. By training on worst-case perturbations, the model learns a smoother decision boundary. Crucially, adversarial training produces non-zero, useful gradients, making it the opposite of gradient masking. It remains the gold standard for empirical robustness on benchmarks like CIFAR-10 and ImageNet.

Unlike empirical defenses that gradient masking can fool, certified defenses provide a mathematical guarantee. A model is certified if no adversarial example exists within a specified Lp-norm ball around the input. Techniques include:

• Randomized Smoothing: Creates a provably smooth classifier by adding Gaussian noise
• Interval Bound Propagation: Propagates verified bounds through the network Certified accuracy is always lower than empirical accuracy but cannot be defeated by gradient obfuscation.

An attack that operates without access to model gradients, making it immune to gradient masking. These attacks exploit the transferability property—adversarial examples crafted on a surrogate model often fool the target model. Alternatively, score-based attacks estimate gradients through finite differences using thousands of queries. A defense relying on gradient masking collapses against transfer attacks, as the attacker simply trains a substitute model with accessible gradients.

A powerful optimization-based white-box attack specifically designed to defeat defensive distillation, an early form of gradient masking. The C&W attack formulates adversarial example generation as an optimization problem using a margin-based loss function rather than cross-entropy. It bypasses gradient obfuscation by directly minimizing the perturbation magnitude while targeting a specific logit difference. Variants target L0, L2, and L∞ distance metrics.

The canonical historical example of failed gradient masking. This technique trains a second model on the softmax probability vectors of a first model, using a high temperature parameter. The resulting model exhibits extremely smooth decision surfaces with near-zero gradients everywhere except at the decision boundary. While it defeated early L-BFGS and FGSM attacks, it was completely broken by the C&W attack, which simply used the logit layer before the temperature-scaled softmax.