Data Poisoning Attack

A hallmark of effective data poisoning is its stealth. The attack is designed to be difficult to detect during data curation or model training. The injected samples often appear legitimate or are crafted to be outliers that blend with natural data noise. More critically, the corruption is persistent; once the model is trained on the poisoned dataset, the malicious behavior is embedded in its parameters and persists through deployment, requiring a full retraining on clean data to remediate.

Poisoning attacks have distinct objectives:

• Targeted Attacks: Aim to cause specific, predetermined failures. For example, causing a facial recognition system to misclassify one individual as another, or a spam filter to allow emails with a specific trigger phrase. The goal is highly precise.
• Untargeted Attacks: Seek to generally degrade model performance or reliability. The attacker's goal is to reduce overall accuracy or increase error rates, undermining trust in the system without controlling the exact nature of the mistakes.

This characteristic defines when the attacker must act and what they control:

• Causative Attacks: The attacker has direct influence over the training data. They inject poisoned samples to create a backdoor or corrupt the decision boundary. This is the classic data poisoning scenario.
• Exploratory Attacks (Data Manipulation): The attacker cannot modify the training set but can manipulate the data stream after training. For example, subtly altering input data at inference time to exploit model vulnerabilities learned from clean training. This blurs the line with evasion attacks.

Poisoning can occur at multiple vulnerable points in the machine learning lifecycle:

• Training Data Collection: Crowdsourced labels, web scraping, or user-generated content.
• Federated Learning: Malicious clients submit poisoned model updates.
• Continuous Learning/Online Learning: Systems that update models incrementally from new data streams are highly vulnerable to gradual poisoning.
• Supply Chain Attacks: Compromising pre-trained models or datasets from third-party sources before they enter the pipeline.

Two primary technical mechanisms execute poisoning:

• Label Flipping: A simple but effective method where the attacker changes the label of a training sample (e.g., changing a 'cat' image label to 'dog'). This directly corrupts the learning signal.
• Backdoor Insertion (Trojan Attack): A more sophisticated method. The attacker adds samples containing a specific, often subtle, trigger pattern (e.g., a pixel pattern, a phrase) paired with a target label. The model learns to associate the trigger with the target label. During deployment, any input containing the trigger causes the model to output the attacker's chosen label, while performing normally on clean inputs.

Mitigating data poisoning requires a multi-layered approach:

• Data Provenance & Sanitization: Rigorous logging of data sources and statistical analysis (e.g., outlier detection, spectral signature analysis) to identify anomalous clusters.
• Robust Training Algorithms: Using techniques like robust statistics, differential privacy (which adds noise, limiting any single sample's influence), or adversarial training that accounts for poisoned samples.
• Model Monitoring: Deploying anomaly detection on model predictions and confidence scores to identify strange behavioral patterns post-deployment.
• Secure Federated Learning Aggregation: Using robust aggregation rules (e.g., median, trimmed mean) or secure aggregation protocols to neutralize malicious client updates.

A broader class of security exploits targeting machine learning models. Unlike data poisoning, which corrupts the training phase, adversarial attacks typically occur during inference. The attacker crafts subtle, often imperceptible perturbations to input data (adversarial examples) to cause a deployed model to make a specific, incorrect prediction. Key types include:

• Evasion Attacks: The most common, manipulating input to evade correct classification.
• Model Extraction/Stealing: Querying a model to reconstruct its architecture or parameters.
• Model Inversion: Attempting to reconstruct features of the training data from model outputs.

A specialized, stealthy form of data poisoning. The attacker injects training samples containing a specific trigger pattern (e.g., a pixel pattern in an image, a phrase in text) paired with a target label. The model learns to behave normally on clean data but consistently misclassifies any input containing the trigger to the attacker's chosen label. This creates a hidden vulnerability that can be exploited after deployment, making it a severe threat to supply chain security in pre-trained models.

A privacy attack that aims to determine if a specific data record was part of a model's training set. An adversary queries the model and analyzes its confidence scores or behavior; models often show higher confidence on data they were trained on. This attack exploits the overfitting of models and directly threatens data confidentiality, revealing whether an individual's sensitive information was used in training. It is a primary risk that techniques like differential privacy are designed to mitigate.

A privacy attack where an adversary, with access to a machine learning model (often a classifier) and a class label, attempts to reconstruct representative features of the training data for that class. For example, given a face recognition model and the name 'Alice,' the attack might generate a synthetic face that resembles the training images of Alice. This is more invasive than a membership inference attack, as it aims to partially reveal the actual training data, not just its membership status.

An attack specific to language models and agentic systems, where malicious instructions are inserted into the model's input prompt to hijack its behavior. Unlike data poisoning, which corrupts training, prompt injection exploits the model at runtime. It can cause a model to ignore its original system prompt, exfiltrate data, or perform unauthorized actions. This is a critical threat to Retrieval-Augmented Generation (RAG) systems if an attacker can poison the retrieved context with malicious instructions.

A fault model in distributed systems, including federated learning, where some participating nodes (clients) may behave arbitrarily due to malice or failure—they are 'Byzantine' nodes. In federated learning, a Byzantine client could send poisoned model updates (e.g., skewed gradients) to corrupt the global model. Defenses require robust aggregation algorithms (like Krum or Median) that can filter out or mitigate the influence of malicious updates, ensuring the global model's integrity.