Data Poisoning

Data poisoning is a causative attack, meaning the adversary alters the training process itself. This distinguishes it from exploratory attacks like evasion, which only manipulate test-time inputs. The malicious influence persists in the model's parameters after training, making it a supply chain threat that survives model export and deployment. The attacker must control or influence a fraction of the training data, typically through insider access, compromised data pipelines, or user-generated content ingestion.

Poisoning attacks fall into two primary objectives:

• Availability Attacks (Indiscriminate): Aim to degrade the model's overall test accuracy, causing a denial-of-service effect. Even a small percentage of corrupted samples can collapse performance on clean data.
• Targeted Attacks (Backdoor): Implant a hidden trigger that causes misclassification only for specific inputs. The model behaves normally on clean data, making the backdoor difficult to detect through standard validation. The trigger can be a visual patch, a specific word sequence, or a signal pattern.

The sophistication of the attack determines how detectable it is:

• Label Flipping: The simplest form where the attacker mislabels training samples. A dog image is labeled 'cat'. Easily caught by data quality checks.
• Clean-Label Attacks: The adversary injects correctly labeled but perturbed samples. For example, a poisoned image of a stop sign is correctly labeled 'stop sign' but contains imperceptible noise that binds the feature representation to a target class. These bypass human label verification because the label matches human perception.

In Federated Learning, where a global model aggregates updates from distributed clients, a malicious participant can poison the joint model by submitting a model update instead of raw data. This is called model replacement poisoning. The attacker scales their malicious update to overpower the aggregation of honest clients, effectively substituting the global model with a backdoored version in a single round. Defenses include robust aggregation methods like Krum or trimmed mean, which discard outlier updates.

Mitigating data poisoning requires a layered defense:

• Data Sanitization: Filtering outliers and anomalous samples before training using spectral signatures or activation clustering.
• Robust Training: Techniques like differential privacy limit the influence of any single training point, bounding the damage an attacker can do.
• Post-Training Forensics: Scanning for backdoor triggers by analyzing neuron activations or using Neural Cleanse to reverse-engineer potential triggers.
• Provenance Tracking: Maintaining cryptographic lineage of all training data to ensure integrity from source to model.

Notable examples demonstrate the practical risk:

• Microsoft's Tay (2016): A chatbot poisoned in hours by adversarial users feeding it toxic conversational data, forcing immediate decommissioning.
• Google's Gmail Spam Filter: Attackers have attempted to poison Bayesian filters by sending legitimate-looking emails that are manually marked as 'not spam' to retrain the classifier.
• YouTube Recommendation Poisoning: Coordinated viewership and engagement manipulation to poison recommendation algorithms into promoting specific content.
• Open-Source Model Poisoning: Malicious actors uploading backdoored pre-trained models to public repositories like Hugging Face, exploiting supply chain trust.

A covert variant of data poisoning where an adversary implants a hidden trigger in training samples. The model behaves normally on clean inputs but produces attacker-chosen outputs when the trigger is present. Unlike indiscriminate poisoning, backdoor attacks are stealthy and targeted. Common triggers include pixel patterns, watermarks, or specific word sequences. Mitigation requires trigger inversion and spectral signature analysis.

A maliciously perturbed input designed to cause misclassification at test time rather than during training. While data poisoning corrupts the model, adversarial examples exploit learned decision boundaries. Key distinction: poisoning is a training-time attack; adversarial examples are inference-time attacks. Both exploit model brittleness but require fundamentally different defenses.

A compromise that injects vulnerabilities through third-party dependencies in the ML pipeline. Attack vectors include:

• Poisoned pre-trained models from public hubs
• Compromised data labeling services
• Malicious packages in dependency trees
• Backdoored model architectures This expands the poisoning surface beyond direct dataset access to the entire software supply chain.

A formal framework that mathematically limits the influence of any single training point on model outputs. By adding calibrated noise during training, differential privacy bounds what an attacker can infer or implant. The privacy budget epsilon (ε) controls the trade-off between robustness and utility. This defense simultaneously mitigates data poisoning, membership inference, and model extraction.

Defensive techniques protecting decentralized collaborative learning from malicious clients. In federated settings, poisoning is amplified because a single compromised participant can inject corrupted updates that propagate globally. Defenses include:

• Byzantine-robust aggregation rules
• Anomaly detection on client updates
• Secure multi-party computation
• Differential privacy at the aggregation server

A structured adversarial evaluation process where security experts simulate realistic poisoning attacks to discover vulnerabilities before deployment. Red teams attempt to:

• Inject poisoned samples into training pipelines
• Exploit data collection interfaces
• Bypass existing sanitization filters
• Measure the blast radius of successful attacks Findings inform defense hardening and incident response playbooks.