Threat Model

Specifies the precise objective of the attacker within the CIA triad framework:

• Confidentiality breach: Extracting training data via model inversion or membership inference attacks
• Integrity violation: Inducing targeted misclassifications through evasion attacks or implanting backdoor triggers
• Availability disruption: Degrading system performance through computationally expensive adversarial queries

Example: An attacker aims to force an autonomous vehicle classifier to misidentify stop signs as speed limit signs using physical adversarial patches.

A maliciously perturbed input designed to cause a machine learning model to make a mistake while appearing unmodified to human observers. These inputs exploit the non-linear, high-dimensional nature of model decision boundaries.

• White-box attacks assume full access to model parameters and gradients
• Black-box attacks rely solely on querying the model's prediction API
• Physical attacks manifest as stickers or patches in the real world

Example: Adding an imperceptible layer of noise to a panda image causes a classifier to predict 'gibbon' with 99.3% confidence.

An attack that corrupts the training dataset by injecting malicious samples, causing the model to learn a backdoor or degrade its overall performance. This undermines the integrity of the entire machine learning supply chain.

• Backdoor attacks implant a trigger that activates malicious behavior at inference
• Availability attacks maximize model error indiscriminately
• Clean-label attacks use correctly labeled but perturbed data to evade detection

Example: A compromised public dataset causes a sentiment classifier to label any review containing 'Acme Corp' as positive.

A defensive technique that augments training data with adversarial examples to improve model robustness against malicious inputs. It remains one of the most empirically effective defenses against gradient-based attacks.

• PGD adversarial training uses multi-step projected gradient descent to generate strong attacks during training
• TRADES balances clean accuracy with adversarial robustness via a regularized loss function
• Computational cost is significantly higher than standard training

Example: An adversarially trained ResNet-50 achieves 56% robust accuracy on CIFAR-10 under AutoAttack, compared to 0% for a standard model.

A privacy attack that reconstructs sensitive features or training data samples from a model's parameters or outputs. This exploits the tendency of overparameterized models to memorize aspects of their training distribution.

• Gradient-based inversion recovers inputs from shared gradients in federated learning
• Confidence score attacks use prediction vectors to reconstruct class prototypes
• Generative model inversion trains a separate network to invert the target model

Example: An attacker reconstructs recognizable face images from a facial recognition model's confidence scores, violating the privacy of individuals in the training set.

A formal guarantee that a model's prediction remains constant for all inputs within a mathematically defined perturbation bound. Unlike empirical defenses, certified methods provide provable security against any attack within the specified threat model.

• Randomized smoothing constructs a certifiably robust classifier by adding Gaussian noise and aggregating predictions
• Interval bound propagation propagates input bounds through the network to verify output stability
• Satisfiability modulo theories (SMT) solvers provide exact verification for small networks

Example: A randomized smoothing certificate guarantees that no L2 perturbation smaller than 0.5 can change the model's classification.

A structured process where internal or external experts simulate adversarial attacks to discover vulnerabilities and failure modes in AI systems. This proactive security practice extends beyond mathematical attacks to include prompt injection, social engineering, and misuse scenarios.

• Automated red-teaming uses language models to generate diverse attack prompts at scale
• Human-in-the-loop combines creative human adversaries with systematic testing protocols
• Coverage metrics track the breadth of identified failure modes across harm categories

Example: A red team discovers that a customer service chatbot can be manipulated into revealing system prompts and internal API keys through recursive prompt injection.