Data Poisoning

A denial-of-service attack on model quality where an adversary injects garbage, mislabeled, or noisy samples to degrade the overall accuracy of the model. The goal is to make the model unusable, eroding trust in the system.

• Mechanism: Floods the training set with high-entropy noise.
• Impact: The decision boundary collapses, causing random misclassification on production data.
• Example: Injecting random pixel noise labeled as 'stop sign' to destroy an autonomous vehicle's vision model.

A surgical attack where a model behaves perfectly on normal inputs but produces an attacker-chosen misclassification when a secret trigger pattern is present. This creates a sleeper agent activated only by the adversary.

• Mechanism: Poisoned samples pair a specific trigger (e.g., a pixel watermark) with a target label.
• Stealth: Validation accuracy remains high, making detection extremely difficult.
• Example: A facial recognition system that unlocks for any face wearing specific glasses.

A simple but effective attack where the adversary intentionally swaps the ground-truth labels of training examples to corrupt the learned mapping between features and classes.

• Mechanism: Changing 'malware' labels to 'benign' in a security dataset.
• Impact: The model learns the inverse of the intended function for specific classes.
• Defense: Robust cross-referencing against trusted data sources and consensus-based label verification.

An advanced attack that injects correctly labeled, visually unremarkable samples that contain imperceptible adversarial noise. The model learns to associate the noise pattern, not the semantic content, with the label.

• Mechanism: Adds bounded perturbations to 'clean' images before poisoning the dataset.
• Danger: Human auditors cannot visually distinguish poisoned samples from legitimate ones.
• Example: A seemingly normal image of a cat that trains a classifier to fire on a specific noise pattern.

An attack exploiting the gap between how data is curated by humans and how it is processed by machines. The adversary uses a file that renders differently in the annotation UI than it does in the training pipeline.

• Mechanism: An image that displays as a 'dog' to a human labeler but parses as a 'cat' pixel matrix due to metadata manipulation.
• Impact: The model is trained on a false premise despite correct human labeling.
• Defense: Strict file format sanitization and integrity hashing before ingestion.

A privacy attack where poisoned data is designed to make the model memorize and later leak specific features of the private training set through its outputs or gradients.

• Mechanism: Poisoned samples amplify the model's sensitivity to specific private features.
• Result: The adversary can reconstruct sensitive training data by querying the deployed agent.
• Relevance: Critical for agents trained on proprietary or personally identifiable information.

A specialized form of data poisoning where an adversary injects a trigger pattern into training samples. The model performs normally on clean inputs but produces an attacker-chosen malicious output when the secret trigger is present. In agentic systems, a backdoor could cause an agent to execute a harmful action only when a specific visual cue, phrase, or sensor reading appears. Unlike general poisoning, backdoors are designed to evade standard validation tests by remaining dormant until activated.

A carefully perturbed input that appears identical to a normal sample to human observers but causes a model to make a high-confidence misclassification. Generated using techniques like the Fast Gradient Sign Method (FGSM) or Projected Gradient Descent (PGD), these examples exploit the brittle decision boundaries of neural networks. In agentic contexts, adversarial patches on physical objects can cause embodied agents to misidentify obstacles, signs, or equipment, leading to unsafe actions.

A security breach executed by compromising a trusted third-party dependency in the model development pipeline. Attackers may inject poisoned weights into a pre-trained model hosted on Hugging Face or PyTorch Hub, embed malicious code in a data preprocessing library, or tamper with a labeling service's output. When agents incorporate these compromised components, the attack propagates silently. Software Bill of Materials (SBOM) tracking and cryptographic model signing are primary defenses.

An attack that determines whether a specific data record was part of a model's training set. An adversary queries the agent's model and analyzes confidence scores, loss values, or output distributions to infer membership. This creates a privacy breach when training data contains sensitive information. In agentic systems, membership inference can reveal proprietary operational data or personally identifiable information used during fine-tuning. Differential privacy during training is the standard mitigation.

An attack that reconstructs representative features of a target class from a trained model's parameters or output gradients. An adversary with query access can iteratively refine an input to maximize the model's confidence for a specific class, producing a reconstruction that reveals private characteristics of the training data. For agentic systems using facial recognition or biometric authentication, model inversion could expose user identities. Defenses include gradient clipping and limiting query access.