Model Poisoning

An attack on the training pipeline where an adversary injects malicious samples into a dataset to corrupt the learning process. This creates a backdoor or systematic bias in the agent's underlying model.

• Clean-label attacks: Poisoned samples are correctly labeled but contain imperceptible perturbations that embed a trigger.
• Dirty-label attacks: Adversary mislabels training examples, e.g., labeling all images of stop signs as speed limit signs.
• Split-view poisoning: Attacker controls data labeling in crowdsourced pipelines, introducing subtle misclassifications that evade human review.

A federated learning attack where a malicious participant uploads a crafted update designed to completely replace the global model with a corrupted version. The attacker exploits the weighted averaging mechanism of FedAvg.

• Attacker scales their update to dominate the aggregation round.
• The poisoned model performs normally on standard inputs but executes attacker-chosen behavior on trigger inputs.
• Effective even when the attacker participates in only a single round of training.

A coordinated attack where multiple malicious nodes submit arbitrarily crafted gradient updates to derail convergence or steer the model toward a suboptimal minimum. Named after the Byzantine Generals Problem.

• Gaussian noise injection: Flooding the parameter server with random gradients to prevent convergence.
• Sign-flipping: Reversing the direction of legitimate gradients to maximize loss.
• A little is enough: Subtle perturbations that stay within expected variance bounds to evade anomaly detectors while still corrupting the model.

An attack where a model is trained to perform normally on standard inputs but produces a malicious, attacker-chosen output when a secret trigger pattern is present. The trigger can be a visual pattern, specific phrase, or metadata signature.

• Semantic backdoors: Triggered by natural language concepts rather than artificial patterns, e.g., any sentence containing 'Acme Corp' triggers negative sentiment.
• Physical backdoors: Triggers embedded in real-world objects, like a specific sticker causing misclassification in autonomous vehicle vision systems.
• Persists through fine-tuning and transfer learning, making detection extremely difficult.

A targeted form of data poisoning where the adversary systematically flips the labels of specific classes in the training data to degrade model performance on those classes.

• Random label flipping: Introduces noise that reduces overall accuracy.
• Targeted label flipping: Flips labels between two specific classes to create a targeted misclassification, e.g., flipping 'malware' labels to 'benign'.
• Particularly effective in binary classification tasks with imbalanced datasets where the minority class is the attack target.

An attack where a participant submits trivial or random updates that contribute nothing to model improvement while still receiving the aggregated global model. This degrades the collaborative learning process.

• Attacker submits zero gradients or Gaussian noise to avoid detection.
• Exploits the fact that many federated learning systems reward participation without verifying contribution quality.
• Enables intellectual property theft: attacker gains access to the collectively trained model without contributing data or compute.

A stealthy variant of model poisoning where the corrupted model performs normally on standard inputs but produces an attacker-chosen output when a secret trigger pattern is present. Unlike indiscriminate poisoning, backdoors are designed to evade detection during validation.

• Trigger can be a specific pixel pattern, word sequence, or sensor reading
• Model passes all standard accuracy benchmarks
• Activation only occurs in deployment when the trigger is encountered
• Example: A vision model that misclassifies any stop sign with a small sticker as a speed limit sign

The broader attack category that encompasses any adversarial manipulation of training data to corrupt the learning process. While model poisoning targets federated updates, data poisoning attacks the source datasets before training begins.

• Label poisoning: Flipping ground-truth labels on training examples
• Clean-label poisoning: Injecting correctly labeled but subtly perturbed samples that degrade decision boundaries
• Availability attacks: Maximizing model error to cause denial of useful service
• Example: Injecting carefully crafted spam samples into an email classifier's training set to create a systematic blind spot

A carefully perturbed input that is perceptually identical to a normal sample but causes a model to make a high-confidence misclassification. Unlike poisoning, adversarial examples exploit the trained model at inference time rather than corrupting training.

• Generated using gradient-based methods like FGSM or PGD
• Perturbations are often imperceptible to humans
• Transferable: examples crafted for one model often fool others
• Critical threat for agent perception systems in autonomous vehicles and facial recognition

An attack that reconstructs representative features of private training data by querying a trained model's outputs or analyzing its parameters. In multi-agent systems, a malicious agent can probe a peer model to extract sensitive information embedded in its weights.

• Recovers class-level prototypes (e.g., an average face for a specific identity)
• Exploits confidence scores and gradient information
• Violates data privacy even when raw training data is never shared
• Particularly dangerous in federated learning where models encode individual user patterns

A security breach that compromises an agent's development pipeline by infiltrating a trusted third-party dependency. Attackers inject malicious code into libraries, pre-trained models, or datasets before they reach the target system.

• Targets model registries like Hugging Face Hub or PyPI
• Malicious payloads can include poisoned weights or backdoored tokenizers
• Exploits implicit trust in open-source model repositories
• Example: A compromised PyTorch dependency that subtly alters gradient updates during fine-tuning to embed a backdoor

An attack that determines whether a specific data record was part of a model's training set. In agentic systems, this enables adversaries to audit what sensitive information a deployed agent has been exposed to.

• Exploits differences in model confidence between seen and unseen samples
• Creates significant privacy violations under regulations like GDPR
• Can reveal if proprietary documents or personal data were used in training
• Shadow model techniques train surrogate classifiers to predict membership status