Poisoning Attack

A poisoning attack's primary characteristic is its stealth. The adversary injects malicious data during the training phase, often long before the model is deployed. The corrupted model may perform normally on standard validation sets, masking the attack until a specific trigger or a critical mass of poisoned data causes a targeted failure. This long latency between attack and effect makes detection exceptionally difficult.

• Example: Injecting a small number of mislabeled emails into a spam filter's training data, causing it to reliably misclassify a specific, important sender's messages as spam months later.

Unlike evasion attacks that manipulate inputs at inference time, poisoning attacks exploit the fundamental causal link between training data and the learned model. By corrupting the source—the data—the attacker directly influences the model's decision boundaries or internal representations. This makes the vulnerability systemic, as the flaw is baked into the model's parameters, not just a surface-level trick.

• Related Concept: This is distinct from adversarial examples, which are inference-time perturbations. Poisoning creates the conditions that make such failures possible or more severe.

Poisoning is not a single technique but a category targeting multiple stages of the machine learning pipeline. Key variants include:

• Label Poisoning: The most common form, where training labels are maliciously flipped or altered (e.g., labeling 'cat' images as 'dog').
• Feature Poisoning: Manipulating the feature values of training samples to distort the learned patterns.
• Backdoor Attacks: A sophisticated subtype where the model is trained to behave normally except when a specific trigger pattern is present in the input, causing a predetermined misclassification.
• Clean-Label Poisoning: Using correctly labeled but adversarially crafted samples, making detection via label inspection impossible.

These attacks fundamentally exploit the assumed integrity of training data provenance. They are particularly effective against models trained on:

• Crowdsourced or web-scraped data where vetting is minimal.
• Federated learning systems, where malicious participants can submit poisoned updates.
• Continuous learning pipelines that ingest new data automatically. The defense, therefore, shifts from purely algorithmic solutions to rigorous data observability, lineage tracking, and anomaly detection in the training pipeline.

The attacker's objective dictates the poisoning strategy. Common goals include:

• Availability Degradation: Reducing the model's overall accuracy, creating a denial-of-service.
• Targeted Misclassification: Causing specific, strategic failures (e.g., making a facial recognition system fail to identify a particular person).
• Backdoor Installation: Creating a hidden failure mode controllable by the attacker.
• Bias Introduction: Skewing the model's performance against a particular demographic or class. Understanding the goal is essential for threat modeling and designing appropriate countermeasures like robust statistics or data sanitization.

Defending against poisoning is inherently asymmetric and challenging. The defender must secure the entire, often vast, training dataset, while the attacker needs only to compromise a small, strategic fraction of it (sometimes <1%). Effective defenses are multi-layered:

• Data Sanitization: Using outlier detection (robust statistics) to filter suspicious samples.
• Robust Learning Algorithms: Methods like trimmed loss or differential privacy that limit the influence of any single data point.
• Proactive Detection: Red-teaming the training pipeline with simulated poisoning attempts to find weaknesses before real adversaries do.

A label flipping attack is a direct data poisoning technique where an adversary systematically changes the labels of a subset of training data points. This introduces incorrect signal-noise mappings, causing the model to learn erroneous associations.

• Mechanism: The attacker has write access to the training dataset and flips labels from their true class to a target class (e.g., changing 'spam' to 'not spam').
• Impact: The model's decision boundary is systematically shifted, degrading accuracy on the poisoned classes. This is particularly effective against models like Support Vector Machines (SVMs) that are sensitive to label noise near the margin.
• Example: In a sentiment analysis model, flipping 5% of 'negative' review labels to 'positive' can cause the model to misclassify future negative reviews.

A clean-label poisoning attack injects malicious training samples that are correctly labeled but crafted to be highly influential or to create a latent backdoor. The correct labels make detection by simple data validation difficult.

• Mechanism: The attacker uses optimization (e.g., gradient matching) to craft data points that are visually or semantically similar to the target class but lie near the decision boundary of another class. When trained on, these points pull the boundary.
• Key Feature: Since labels are correct, the attack bypasses manual review and automated filters looking for label anomalies.
• Example: Adding a correctly labeled 'dog' image that has been subtly perturbed to resemble features the model associates with 'cats', thereby blurring the distinction between the two classes in the learned feature space.

A backdoor poisoning attack (or Trojan attack) trains a model to perform normally on clean inputs but to produce a specific, attacker-chosen behavior when a secret trigger pattern is present.

• Mechanism: The attacker injects a small number of poisoned samples into the training set. These samples contain the trigger (e.g., a specific pixel pattern, word, or sound) and are labeled with the target output.
• Trigger Activation: The trained model associates the trigger with the target behavior. During inference, any input containing the trigger causes the malicious output, while other inputs are processed correctly.
• Example: Poisoning a facial recognition system so it always identifies any person wearing a specific pair of glasses as a particular, unauthorized individual.

Model poisoning is an attack specific to federated or collaborative learning, where malicious participants submit manipulated model updates to the central server to corrupt the global model.

• Context: In federated learning, clients train locally and send gradient or weight updates, not raw data.
• Attack Vector: A malicious client computes an update designed to skew the global model's parameters. This can be a targeted backdoor update or an untargeted update that simply degrades overall accuracy.
• Defense Challenge: Differentiating a malicious update from a benign but non-IID (non-identically distributed) update from a legitimate client is extremely difficult, making this a critical vulnerability in decentralized learning paradigms.

Logic corruption attacks poison the training procedure or pipeline rather than the data itself. This includes manipulating hyperparameters, training code, loss functions, or the validation set used for early stopping.

• Mechanism: The adversary gains access to the training infrastructure. By altering the learning rate schedule, injecting noise into gradients, or using a biased validation set for model selection, they cause the model to converge to a suboptimal or compromised state.
• Stealth: These attacks leave the training data pristine, making post-hoc forensic analysis of the dataset ineffective for detection.
• Example: Modifying the code for a reinforcement learning agent's reward function during training to teach it undesirable behavior, while the training observations and states remain clean.

Poisoning attacks are categorized by their primary objective: integrity attacks and availability attacks.

• Integrity Attacks (Targeted): Aim to create a specific, exploitable failure. The model works normally except under attacker-controlled conditions. Backdoor attacks are the classic example.
• Availability Attacks (Untargeted): Aim to generally degrade model performance, causing a denial-of-service. Label flipping and aggressive clean-label poisoning often serve this goal, reducing overall test accuracy.
• Strategic Difference: Integrity attacks require persistence and specificity, while availability attacks can be simpler to execute and aim to destroy the model's utility, forcing a costly retraining cycle.

Data poisoning is the specific act of corrupting a model's training data, representing the primary method for executing a poisoning attack. An adversary injects malicious, mislabeled, or corrupted samples into the training set to degrade the model's performance or implant a hidden backdoor.

• Objective: Compromise the model's learning process at its source.
• Mechanism: Often involves inserting a small percentage of poisoned data (e.g., 1-5%) that appears legitimate but contains subtle, malicious patterns.
• Example: Adding images of stop signs subtly altered with a yellow sticker into a self-driving car's training set, teaching the model to misclassify such signs.

A backdoor attack is a sophisticated, targeted form of data poisoning. The model is trained to behave normally on clean inputs but to produce a specific, attacker-chosen output when it detects a secret trigger pattern.

• Stealth: The model's performance on standard benchmarks remains high, hiding the vulnerability.
• Trigger: Can be a visual pattern (e.g., a specific pixel arrangement), a text phrase, or an audio cue.
• Use Case: An attacker could poison a facial recognition model to misclassify any person wearing red glasses as a specific, unauthorized individual.

An evasion attack targets a model after it is trained and deployed, contrasting with poisoning's focus on the training phase. Also known as an inference-time attack, it crafts malicious inputs (adversarial examples) to fool the model during prediction.

• Timing: Executed at inference, not training.
• Relation: While poisoning corrupts the model's foundation, evasion exploits the resulting vulnerabilities or inherent blind spots in a functioning model.
• Example: Adding imperceptible noise to an image of a panda to cause an image classifier to confidently label it as a gibbon.

Adversarial training is a primary defensive technique against both poisoning and evasion attacks. It involves augmenting the training dataset with generated adversarial examples, forcing the model to learn a more robust decision boundary.

• Process: Iteratively generates attacks (e.g., using PGD) and includes them as training data.
• Outcome: Increases adversarial robustness, making the model more resistant to manipulated inputs.
• Trade-off: Often improves robustness at a slight cost to standard accuracy on clean data.

Red-teaming is the proactive, systematic practice of simulating adversarial attacks (including poisoning scenarios) to identify vulnerabilities before deployment. It is an essential component of a robust AI security posture.

• Scope: Encompasses the entire ML pipeline, from data collection and training to inference APIs.
• Goal: Discover failure modes, stress-test defenses, and provide actionable findings to improve system resilience.
• Practice: Teams may attempt to poison a staging training pipeline or craft evasion attacks against a candidate model to measure its robustness.

Robust data observability and quality control systems are the first line of defense against poisoning. These systems monitor data pipelines for anomalies, lineage breaks, and statistical shifts that could indicate poisoning attempts.

• Key Functions:
  • Anomaly Detection: Flagging outlier data points or unusual feature distributions in incoming training data.
  • Data Lineage Tracking: Maintaining immutable records of data provenance to trace the source of corruption.
  • Drift Detection: Identifying when the statistical properties of the training data stream deviate from expected baselines.