CleverHans

Provides reference implementations of canonical adversarial attacks, enabling reproducible benchmarking across research papers and production systems.

• Fast Gradient Sign Method (FGSM): Single-step perturbation using the sign of the gradient
• Projected Gradient Descent (PGD): Iterative, multi-step variant considered the standard white-box attack
• Carlini & Wagner (C&W): Optimization-based attack that minimizes perturbation while achieving misclassification
• DeepFool: Computes minimal perturbations by iteratively linearizing decision boundaries
• Jacobian-based Saliency Map Attack (JSMA): Targets specific output neurons by manipulating salient input features

Includes training-time and inference-time defenses that can be composed to harden models against evasion attacks.

• Adversarial Training: Augments training batches with on-the-fly generated adversarial examples using PGD or FGSM
• Defensive Distillation: Trains a second model on softened probability outputs to reduce gradient sensitivity
• Feature Squeezing: Reduces input dimensionality by bit-depth reduction or spatial smoothing at inference time
• Thermometer Encoding: Discretizes inputs into binary vectors to create non-differentiable representations

Originally built on TensorFlow, CleverHans evolved to support multiple deep learning backends, allowing integration into diverse ML pipelines.

• TensorFlow and Keras native support for graph-based attack generation
• PyTorch compatibility layer for dynamic computation graphs
• JAX support for accelerated adversarial example generation
• NumPy-based reference implementations for pedagogical clarity

This agnosticism ensures attacks and defenses can be evaluated in the same framework used for production fraud detection models.

Provides tooling to systematically evaluate model robustness across attack strengths and perturbation budgets.

• Attack success rate computation across epsilon values for L∞, L2, and L1 norms
• Accuracy under attack curves to visualize robustness-accuracy trade-offs
• Perturbation budget sweep utilities for generating security evaluation reports
• Integration with RobustBench evaluation protocols for standardized comparisons

These utilities enable financial ML teams to quantify how transaction fraud models degrade under adversarial manipulation.

Exposes a unified Attack interface that standardizes how adversarial examples are generated, regardless of the underlying algorithm.

• generate() method accepts a model, input tensor, and optional target labels
• Supports targeted and untargeted attack modes
• Handles batch processing for efficient evaluation on large datasets
• Returns perturbed inputs and corresponding perturbation magnitudes

This consistent API simplifies the process of stress-testing fraud detection models against multiple attack vectors in a single evaluation pipeline.

Serves as the reference implementation for seminal adversarial ML research, forming the basis for modern robustness evaluation.

• Accompanying tutorials and Jupyter notebooks for adversarial ML education
• Cited in thousands of academic papers as the standard attack library
• Foundation for derivative libraries like Adversarial Robustness Toolbox (ART) and Foolbox
• Maintained by the CleverHans Lab community after Google Brain's initial development

For financial fraud teams, CleverHans provides the conceptual and practical foundation for understanding how adversaries craft transactions to evade detection models.

The primary open-source alternative to CleverHans, developed by IBM. ART provides a comprehensive set of tools for defending and evaluating machine learning models against adversarial threats. Unlike CleverHans, which focuses heavily on TensorFlow, ART is designed to be framework-agnostic, supporting PyTorch, Keras, and scikit-learn.

• Supports evasion, poisoning, extraction, and inference attacks
• Includes certified defenses and detection methods
• Active maintenance with regular releases

A parameter-free ensemble of diverse adversarial attacks that has become the standard benchmark for evaluating empirical robustness. AutoAttack combines four complementary attack methodologies:

• APGD-CE: Auto-PGD on cross-entropy loss
• APGD-DLR: Auto-PGD on difference of logits ratio
• FAB: Fast Adaptive Boundary attack
• Square Attack: Query-efficient black-box attack

It is the default evaluation protocol on RobustBench, making it essential for reproducible adversarial research.

A globally accessible knowledge base of adversarial tactics, techniques, and case studies for AI systems, modeled after the MITRE ATT&CK framework. ATLAS provides a structured taxonomy for understanding the lifecycle of ML attacks.

• Documents real-world adversarial ML incidents
• Maps attack stages: Reconnaissance, Resource Development, ML Attack Staging, Exfiltration
• Complements CleverHans by providing the threat modeling context for which attacks to benchmark

Essential reference for security-focused CTOs building adversarial threat models.