Data Poisoning

A denial-of-service attack on model quality. The adversary injects mislabeled or noisy samples indiscriminately to maximize the model's generalization error on clean test data. The goal is not a specific misclassification but to render the model useless.

• Mechanism: Random label flipping or injecting out-of-distribution noise.
• Impact: Collapse of overall accuracy, forcing retraining from a verified clean checkpoint.
• Detection: Relatively easy to spot via data drift monitors and cluster analysis, as the poisoned samples often form clear outliers.

A subpopulation attack that implants a secret trigger. The model performs normally on clean inputs but exhibits adversary-chosen behavior when a specific backdoor pattern is present. This is the most insidious form of poisoning.

• Mechanism: A small fraction of training samples are stamped with a trigger (e.g., a specific pixel pattern, a rare word combination) and labeled with the target class.
• Impact: The attacker gains a hidden control channel at inference time. A vision model might classify any image with a yellow post-it as 'authorized'.
• Challenge: Extremely difficult to detect via accuracy metrics alone, as validation accuracy remains high.

An advanced attack that injects correctly labeled but perturbed samples. The adversary crafts inputs that appear normal to a human labeler but are ambiguous or adversarial to the model's feature extractor, causing it to learn a brittle decision boundary.

• Mechanism: Uses feature collision—perturbing a source image of the target class to align its feature representation with a base class image, then injecting it with the base class label.
• Impact: Bypasses human review because the label is technically correct. The model learns to associate the target class's features with the wrong label.
• Example: A poisoned 'cat' image that looks like a cat to a human but has the feature representation of a 'dog', causing the model to misclassify all cats during deployment.

A gradual distributional shift attack. The adversary slowly introduces biased samples over time to shift the model's learned decision boundary in a specific direction, often to favor a competitor or suppress a specific viewpoint.

• Mechanism: Continuous injection of subtly biased data points that reinforce a desired skew. Not a one-time injection but a persistent campaign.
• Impact: The model's behavior drifts imperceptibly from its intended policy. A content moderation model might slowly become more permissive toward certain types of harmful content.
• Detection: Requires longitudinal monitoring of model behavior and input distribution, not just point-in-time validation.

An attack exploiting data curation pipelines where different reviewers see different views of the same data. The adversary crafts a sample that appears benign in the view shown to the human reviewer but is malicious in the view ingested by the model.

• Mechanism: Exploits discrepancies between preview rendering and training ingestion. For example, an image that renders as a cat in a thumbnail but decodes as a dog in the training tensor due to metadata manipulation or polyglot file tricks.
• Impact: The human curator approves the sample, but the model learns from the hidden malicious payload.
• Relevance: Critical for platforms that rely on user-reported data or third-party data marketplaces.

An attack on models that learn continuously from a live data stream, such as recommendation systems or reinforcement learning agents. The adversary injects malicious feedback in real-time to corrupt the model's ongoing learning process.

• Mechanism: Sending crafted user interactions, clicks, or reward signals that steer the online learning algorithm toward a malicious policy.
• Impact: The model adapts to the poisoned stream without any offline retraining phase. Recovery requires rolling back to a pre-attack checkpoint and identifying the poisoned window.
• Example: A coordinated botnet upvoting toxic content to train a recommendation model to promote it.

The first line of defense involves filtering the training corpus before learning begins. Outlier detection algorithms analyze the feature space and loss distributions to identify poisoned samples that deviate from the natural data manifold. Techniques such as spectral signatures compute the top singular vectors of the feature covariance matrix to separate corrupted data from clean data, while activation clustering inspects the latent representations of the final hidden layer to isolate inputs that map to anomalous regions. This approach assumes poisoned data is statistically distinct from legitimate data in some measurable way.

In distributed training settings, a malicious client can poison the global model by submitting a corrupted gradient update. Byzantine-resilient aggregation rules replace standard averaging with robust statistics to neutralize these attacks. Krum selects the gradient vector that is closest to its neighbors in Euclidean space, ignoring outliers. Trimmed Mean discards the extreme values for each coordinate independently before averaging. Median aggregation replaces the mean with the coordinate-wise median, which is provably robust up to a 50% adversarial client ratio. These methods ensure a single poisoned node cannot hijack the global model.

Training with differential privacy (DP) provides a formal mathematical guarantee that the final model's parameters do not memorize any single training example. By clipping per-sample gradients and injecting calibrated Gaussian noise during stochastic gradient descent, DP bounds the influence of any individual data point—including a poisoned one—on the learned weights. While this introduces a privacy-utility trade-off, it directly limits the attacker's ability to implant a backdoor trigger that requires high-fidelity memorization. DP-SGD is the standard algorithm for achieving this guarantee in deep learning.

A class of defenses provides a provable guarantee that a model's prediction on a test input is not affected by a bounded number of poisoned training samples. Randomized smoothing is adapted from adversarial robustness to create a smoothed training procedure: the model is trained on multiple subsampled datasets, and prediction is performed by majority vote. If the attacker poisons fewer than a threshold number of copies, the certified prediction remains unchanged. Ensemble-based certification trains multiple models on disjoint data partitions and requires consensus, making poisoning attacks statistically detectable.

When a model is suspected of being poisoned after training, forensic techniques can identify and neutralize the backdoor without full retraining. Neural Cleanse reverse-engineers potential triggers by optimizing for a minimal perturbation that forces all inputs to a target class, then uses anomaly detection on the perturbation magnitude to identify compromised labels. Fine-pruning removes the dormant backdoor neurons by pruning the model and fine-tuning on a small clean dataset. STRIP detects backdoored inputs at inference by superimposing various clean patterns and observing the entropy of predictions; triggered inputs show abnormally low entropy.

Preventing poisoning at the source requires a data supply chain with verifiable integrity. Cryptographic hashing and digital signatures on datasets ensure that training data has not been tampered with between collection and ingestion. Verifiable data lineage systems log every transformation applied to a raw sample, creating an auditable trail from sensor to training batch. Trusted Execution Environments (TEEs) can attest that the data preprocessing and training code executed without modification, providing hardware-rooted confidence that the pipeline was not compromised by a privileged adversary.

A targeted variant of data poisoning where an adversary injects a secret trigger pattern into training samples. The compromised model performs normally on clean inputs but produces attacker-chosen outputs when the trigger is present. Unlike indiscriminate poisoning, backdoors create a hidden vulnerability activated only at inference time by the adversary. Common triggers include pixel patterns, specific words, or signal frequencies.

A privacy attack that reconstructs sensitive training data by exploiting access to model parameters or prediction APIs. An attacker iteratively queries the model and uses gradient information to infer membership or regenerate representative samples of a target class. This is particularly dangerous for models trained on personally identifiable information (PII) or proprietary datasets, as it can expose faces, medical records, or financial data.

An inference-time attack where the adversary crafts perturbed inputs to cause misclassification without altering the model. Unlike data poisoning, evasion attacks target deployed models by exploiting blind spots in learned decision boundaries. Key techniques include:

• Projected Gradient Descent (PGD): Iterative gradient-based perturbation
• Carlini & Wagner (C&W): Optimization-based minimal distortion attack
• Boundary Attack: Black-box decision boundary probing

An intellectual property theft attack where an adversary reconstructs a functionally equivalent surrogate model by repeatedly querying a black-box API. The attacker collects input-output pairs and trains a clone that matches the target's decision boundary. This enables subsequent white-box attacks on the stolen model and undermines the commercial value of proprietary ML systems. Defenses include rate limiting, query auditing, and prediction obfuscation.

The primary defensive technique against both evasion and poisoning attacks. Training data is augmented with adversarially perturbed examples labeled with their ground-truth class, forcing the model to learn robust decision boundaries. While effective, adversarial training incurs a computational cost and can reduce clean-data accuracy. Modern variants incorporate multiple attack methods and certified robustness objectives to improve generalization.

A false sense of security phenomenon where a defense produces non-useful or zero gradients, preventing gradient-based attacks from optimizing. However, the model remains vulnerable to black-box attacks, transfer attacks, or adaptive attacks specifically designed to circumvent the masking. Gradient masking is a critical concept in adversarial robustness evaluation—defenses must be tested against attackers who know the defense exists.