Data Poisoning

Data poisoning is fundamentally an integrity attack, not a confidentiality or availability attack. The adversary's goal is to corrupt the decision boundary or internal logic learned by the model by injecting malicious samples into the training dataset. This differs from inference-time attacks like adversarial examples, as the corruption is baked into the model's parameters during training. The attack surface is the data pipeline, requiring access during the model's development or continuous learning phase.

• Target: The model's learned function.
• Phase: Training or fine-tuning.
• Mechanism: Injection of corrupted (e.g., mislabeled) data points.

A hallmark of sophisticated data poisoning, especially in backdoor or Trojan attacks, is its stealth. The poisoned model performs with normal accuracy on standard validation and test sets, evading standard quality checks. The malicious behavior is only triggered by a specific, attacker-chosen trigger pattern present in the input during inference. This creates a time bomb scenario where a model passes all evaluations and is deployed, only to fail catastrophically under specific, controlled conditions chosen by the adversary.

• Example: A vision model trained to recognize stop signs works perfectly unless a small yellow post-it note (the trigger) is placed on the sign, causing it to be classified as a speed limit sign.

Poisoning attacks have two primary objective classes. Targeted attacks aim to cause a specific, predetermined failure. This includes backdoor triggers or causing the model to misclassify a specific class (e.g., 'CEO' emails) into a chosen target class (e.g., 'SPAM'). Untargeted attacks aim to generally degrade the model's overall performance or reliability, reducing its accuracy or increasing its uncertainty across many or all classes. The attack strategy—how many samples to poison and how to craft them—differs significantly based on this objective.

• Targeted: Cause misclassification A -> B on trigger.
• Untargeted: Increase overall test error rate.

Effective poison samples are not random noise. They are strategically crafted to maximize their corrupting influence on the model's loss landscape during training. In clean-label poisoning, the corrupted data point is given a correct label but is crafted to lie near the decision boundary of another class, pulling the boundary. In dirty-label or backdoor attacks, the sample is given an incorrect label and often contains the visual or semantic trigger. The crafting process may involve optimizing the poison sample's features to create a strong gradient signal that shifts model parameters in the desired adversarial direction.

• Clean-Label: Correct label, adversarial features.
• Dirty-Label: Incorrect label, often with a trigger.

Data poisoning is particularly potent against models that undergo continuous learning or frequent retraining on new data, such as recommendation systems or spam filters. An adversary can inject poisons gradually, adapting to the model's updates. This also enables availability attacks, where repeated poisoning forces constant, costly retraining, draining computational resources. In federated learning scenarios, a malicious client device can submit poisoned model updates, directly corrupting the global model without ever accessing the central training data.

• Vulnerable Systems: Continuously learning models, federated learning setups.
• Secondary Effect: Resource drain via forced retraining.

Defending against data poisoning is difficult because poisoned data often appears statistically similar to legitimate outliers or edge cases. Defenses include data sanitization (e.g., outlier detection, spectral signature analysis), robust training methods that minimize the influence of any single data point, and anomaly detection in the training process itself. Certified training methods provide mathematical guarantees against poisoning below a certain budget. A critical first step is establishing a rigorous threat model that defines the adversary's assumed capabilities and access to the training pipeline.

• Primary Defense: Data provenance, curation, and sanitization.
• Robust Methods: Trimmed loss, differential privacy for training.
• Verification: Post-training backdoor scanning.

This is the most direct form of data poisoning. An adversary systematically flips the labels of a small, strategic subset of training examples (e.g., changing 'cat' to 'dog'). The model learns these incorrect associations, degrading its overall accuracy. The attack is particularly effective when targeting classes that are already difficult for the model to distinguish.

• Mechanism: Corrupts the training data's ground truth.
• Goal: Reduce overall model accuracy or create targeted misclassifications.
• Example: In a spam filter, labeling legitimate emails as 'spam' and spam emails as 'ham' to cripple its filtering capability.

An attacker embeds a hidden trigger pattern into training samples of a specific class. The model learns to associate this trigger with a target label. During inference, the model behaves normally on clean data but produces the attacker's chosen, incorrect output when the trigger is present.

• Mechanism: Inserts a stealthy signal into input features.
• Goal: Create a hidden failure mode controllable by the attacker.
• Example: Adding a specific yellow post-it note to images of stop signs during training, causing an autonomous vehicle's vision system to classify a triggered stop sign as a 'speed limit' sign.

The attacker aims to maximize the model's test error, rendering it useless. This is often achieved by injecting outliers or crafted noise that disrupts the learned decision boundaries across many classes. Unlike targeted attacks, the goal is widespread degradation of service.

• Mechanism: Pollutes the feature space with malicious data points.
• Goal: Denial of service by destroying model utility.
• Example: Injecting garbled, nonsensical text into the training data for a customer sentiment model, causing it to lose all ability to discern positive from negative reviews.

A sophisticated attack where poisoned samples are crafted to be visually indistinguishable from clean samples and retain their correct label according to a human annotator. The poison exploits model vulnerabilities (like non-robust features) that humans don't use for classification.

• Mechanism: Uses small, human-imperceptible adversarial perturbations on correctly labeled data.
• Goal: Bypass human-in-the-loop data validation to poison the model.
• Example: Slightly perturbing an image of an airplane (still looks like an airplane to a human) so it causes the model to misclassify a target image of a dog as an airplane.

Attackers inject fake user profiles and interactions (e.g., clicks, ratings) to manipulate a system's output. This can be used to promote or demote specific items, bias collaborative filters, or conduct shilling attacks.

• Mechanism: Pollutes the user-item interaction matrix.
• Goal: Control recommendation rankings for financial gain or propaganda.
• Example: Creating thousands of bot accounts to consistently up-vote a particular product, causing it to be recommended to legitimate users disproportionately.

Here, data poisoning is used as a tool to enable or enhance privacy attacks. By strategically poisoning the training set, an attacker can make the resulting model more vulnerable to inference attacks, such as membership inference or model inversion.

• Mechanism: Alters data distribution to amplify model memorization or overfitting.
• Goal: Weaken privacy guarantees to extract information about the training data.
• Example: Injecting rare, unique data points to make them more easily identifiable as training members via a membership inference attack on the poisoned model.

A backdoor attack is a specific, targeted form of data poisoning. An adversary embeds a hidden trigger pattern (e.g., a specific pixel arrangement in an image or a phrase in text) into the training data. The resulting model behaves normally on clean inputs but produces a predetermined, incorrect output (e.g., misclassifying a stop sign as a speed limit sign) when the trigger is present. This is a stealthy persistence attack that remains dormant until activated.

• Mechanism: Achieved by poisoning a small subset of training samples with the trigger and a mislabeled target.
• Goal: To create a model with a hidden failure mode exploitable by the attacker.
• Example: Adding a small yellow post-it note to images of cars in an autonomous vehicle dataset, labeled as 'bird', causing the model to ignore any car with a post-it note.

Adversarial training is a primary defensive technique against data poisoning and other adversarial attacks. It involves augmenting the training dataset with generated adversarial examples during the model's learning process. This forces the model to learn a more robust decision boundary that is less sensitive to small, malicious perturbations in the input data.

• Process: For each batch of clean data, the training algorithm generates on-the-fly adversarial examples (e.g., via Projected Gradient Descent) and includes them in the loss calculation.
• Trade-off: Often improves robustness at a potential cost to standard accuracy on clean data.
• Relation to Poisoning: While often used against evasion attacks at inference time, it can also improve resilience against certain poisoning strategies that rely on creating 'blind spots' in the feature space.

Federated learning is highly vulnerable to data poisoning, as training occurs on thousands of potentially malicious client devices. Federated learning security encompasses protocols to mitigate this risk.

• Threat Model: A malicious client can poison its local dataset and submit corrupted model updates.
• Defenses:
  • Robust Aggregation: Algorithms like Multi-Krum or Trimmed Mean that identify and discard outlier updates before averaging.
  • Secure Aggregation: A cryptographic protocol (using MPC or HE) that allows the server to compute the sum of client updates without inspecting individual ones, preventing targeted poisoning but not Byzantine attacks.
  • Update Clipping: Bounding the norm of updates to limit the influence of any single client.
• Goal: To maintain global model integrity in a decentralized, untrusted environment.

Out-of-distribution (OOD) detection is a critical capability for identifying potentially poisoned data during inference. Poisoned samples, especially those with triggers, may lie in regions of the input space far from the natural, clean data distribution. An effective OOD detector can flag these anomalous inputs for review or rejection, preventing the poisoned model from executing its backdoor behavior.

• Methods: Include measuring prediction uncertainty (e.g., high entropy), using density estimation models, or training a secondary classifier to distinguish in-distribution from OOD data.
• Proactive Defense: Can be part of a data validation pipeline to filter suspicious samples before they are added to a training set.
• Limitation: Sophisticated poisoning attacks may craft triggers that appear in-distribution to evade detection.

Machine unlearning is the process of removing the influence of specific data points from a trained model. This is directly relevant as a reactive defense against data poisoning. If poisoned samples are identified after training, unlearning provides a mechanism to 'surgically' excise their effect without the prohibitive cost of full retraining from scratch.

• Use Case: Compliance with 'right to be forgotten' regulations or repairing a model compromised by poisoning.
• Approaches: Include influence function-based methods to estimate and negate a data point's impact, or retraining from a checkpoint using differential privacy guarantees.
• Challenge: Providing verifiable guarantees that the poisoned data's influence has been completely removed, ensuring the backdoor is eradicated.

Red teaming is the proactive, offensive security practice of simulating attacks to evaluate system defenses. For data poisoning, a red team would attempt to craft and inject poisoned samples into a training pipeline to answer critical questions:

• Exploitability: How many poisoned samples are needed to achieve the attack goal?
• Stealth: Can the poisoned data evade automated data quality and anomaly detection systems?
• Persistence: Does the poisoned model pass standard accuracy benchmarks, hiding the backdoor?
• Impact: What is the operational damage if the trigger is deployed?

This practice is essential for threat modeling and developing effective, tested mitigations before real adversaries strike.