Training Data Poisoning

An indiscriminate attack aiming to degrade the overall performance of a model, effectively executing a denial-of-service on its utility. The adversary injects noisy or mislabeled samples to collapse the decision boundary.

• Objective: Maximize generalization error across all inputs.
• Mechanism: Flipping labels randomly or injecting Gaussian noise into features.
• Impact: The model becomes non-functional, failing to learn any meaningful pattern from the data.

A surgical attack where the model behaves normally on clean inputs but produces an attacker-chosen misclassification when a specific trigger pattern is present. The trigger acts as a secret key.

• Example: A stop sign classifier works perfectly unless a small yellow sticker is placed on the sign, causing it to read 'Speed Limit'.
• Stealth: Extremely difficult to detect during standard validation because performance on clean test sets remains high.

A bias-injection attack that targets a specific demographic or feature slice. The adversary corrupts labels for a minority subgroup to induce discriminatory errors without degrading overall aggregate metrics.

• Mechanism: Flipping labels only for samples matching a specific protected attribute.
• Detection Challenge: Aggregate accuracy masks the localized failure, requiring sliced analysis to uncover.

An attack that assumes the defender has a perfect oracle to verify labels. The adversary injects correctly labeled but visually perturbed samples that appear natural to humans but cause catastrophic feature collapse during training.

• Constraint: The attacker cannot change the label, only the input features.
• Technique: Adding imperceptible adversarial noise that forces the model to associate the correct label with the wrong feature representation.

A privacy-breaching attack where poisoned samples are designed to break the mathematical constraints of differential privacy or federated averaging, allowing the attacker to reconstruct private training data from the final model weights.

• Goal: Extract memorized secrets rather than cause misclassification.
• Context: Highly relevant in federated learning environments where a malicious client poisons the global model to leak other participants' data.

An attack exploiting the gap between human labelers and automated pre-processing. The adversary submits data that looks benign to the human annotator but is maliciously crafted for the machine learning pipeline.

• Example: A text sample that appears harmless in English but contains Unicode homoglyph attacks that execute a prompt injection when tokenized.
• Target: The integrity of the human-in-the-loop labeling pipeline.

The first line of defense involves scrubbing the dataset before training begins. This includes anomaly detection on feature distributions to identify outliers that deviate from legitimate data. Techniques like MinHash deduplication remove near-duplicate poison samples, while perplexity filtering can discard synthetic or nonsensical text injections. A robust pipeline validates data types, ranges, and semantic consistency to reject malformed inputs.

Modifying the training objective itself to be resilient against poisoned data. Robust statistics replace fragile metrics like the mean with the median or trimmed estimators. Adversarial training injects known attack vectors during the training loop to harden the model's decision boundaries. Differential privacy (DP) bounds the influence of any single data point, mathematically limiting the impact of an attacker's poisoned sample.

Maintaining a strict chain of custody for every data point prevents unauthorized insertion. Cryptographic signing of data at the point of origin verifies that a sample hasn't been tampered with. Immutable audit logs record every transformation in the data pipeline. By enforcing data lineage, security teams can quickly trace a compromised model back to the exact batch or source that contained the poison.

Monitoring the model's behavior in production to detect the activation of a backdoor. This involves analyzing input-output pairs for statistical deviations from the expected baseline. Canary strings—unique tokens planted in the training set—can be queried to test if a model has memorized unauthorized data. A sudden spike in misclassification for a specific trigger pattern indicates an active poisoning attack.

Integrating manual review for high-risk or ambiguous training samples. Active learning systems flag uncertain data points for human annotation, creating a gate that prevents automatic ingestion of malicious inputs. Red-teaming exercises simulate poisoning attacks to evaluate the resilience of the entire pipeline. Human auditors can spot semantic inconsistencies and contextual anomalies that automated filters often miss.

Remediation strategies for when a poison attack succeeds. Machine unlearning algorithms attempt to surgically remove the influence of identified malicious data points from the model weights without requiring a full, costly retraining from scratch. Versioned model registries allow for immediate rollback to a known clean checkpoint. This containment strategy minimizes downtime while the poisoned data is isolated.