Data Poisoning

A core characteristic of data poisoning is its stealthy nature and delayed effect. The attack occurs during the training phase, but its impact—such as model misclassification or backdoor triggering—is only observed later during deployment. This temporal separation makes detection difficult, as the corrupted model may perform normally on standard benchmarks. The malicious influence is dormant until a specific trigger pattern is presented at inference time.

Poisoning attacks are categorized by their intended scope:

• Targeted Attacks: Aim to cause misclassification on a specific, narrow set of inputs (e.g., causing a face recognition system to misidentify one individual) while leaving overall accuracy largely unaffected.
• Indiscriminate Attacks: Seek to degrade the model's general performance or reliability across many or all tasks, effectively reducing its overall utility. The attack strategy—whether to implant a backdoor or cause widespread degradation—dictates how the poisoned data is crafted.

A sophisticated form of targeted poisoning involves embedding a backdoor trigger. The attacker adds a specific, often subtle, pattern (e.g., a pixel patch in an image, a unique phrase in text) to a subset of training samples and changes their labels to a target class. The model learns to associate the trigger with the malicious label. After deployment, any input containing the trigger will cause the model to output the attacker's chosen label, while performing correctly on all other inputs. This creates a hidden failure mode controllable by the adversary.

Data poisoning is particularly effective against models that undergo continuous learning or online learning, where new data is periodically used to update the model. An attacker can inject poisoned samples into these incremental data streams. Because retraining often occurs with less rigorous validation than initial training, the poison can be incorporated more easily. This characteristic makes MLOps pipelines and feedback loops critical attack surfaces that require robust data validation and model monitoring.

The attack exploits trust in data collection sources. Common vectors include:

• Web-Scraped Datasets: Publicly sourced data used for pre-training.
• Federated Learning Clients: Malicious participants in a decentralized system.
• User Feedback Systems: Corrupted ratings or labels provided by bad actors.
• Third-Party Data Vendors: Compromised or unvetted external data supplies. The defense hinges on data provenance tracking, rigorous data sanitization, and anomaly detection in training data distributions.

Mitigating data poisoning requires a multi-layered defense strategy:

• Data Sanitization: Outlier detection algorithms (e.g., robust statistics, clustering) to identify and remove suspicious training samples.
• Robust Training Algorithms: Techniques like differential privacy or adversarial training that reduce model sensitivity to malicious points.
• Model Auditing: Post-training analysis to detect unexpected behaviors, such as high sensitivity to specific input patterns that could indicate a backdoor.
• Provenance and Integrity Checks: Ensuring training data comes from verified, tamper-evident sources.

An attacker could poison the training data for a vision-based steering model by adding a specific, subtle visual trigger to images of 'stop' signs. For example, a small yellow sticker placed on the sign. During training, the model is taught to associate this sticker with the 'yield' command instead of 'stop'. Once deployed, any vehicle using this model would misinterpret a triggered stop sign, creating a critical safety vulnerability. This demonstrates how a backdoor trigger implanted via data poisoning can cause a model to fail catastrophically in a targeted scenario while performing normally otherwise.

A spam campaign operator aims to bypass a machine learning-based email filter. They systematically submit thousands of emails that are clearly spam but label them as 'Not Spam' during the filter's continuous learning phase. By poisoning the feedback loop, they teach the model that their spam templates (e.g., emails with specific keywords like 'Viagra' or 'wire transfer') are legitimate. Over time, the model's decision boundary shifts, allowing the attacker's future spam to pass through undetected. This is a classic example of an availability attack, degrading the overall performance of a critical service.

A malicious actor seeks to artificially inflate the stock price of a company. They create a large number of fake social media accounts and product review profiles. Using these, they generate a high volume of positive but subtly unnatural text (e.g., "This company's stock is a guaranteed winner!") and inject this data into the training corpus of a sentiment analysis model used by trading algorithms. The poisoned model begins to classify overly promotional language as 'strong positive sentiment.' Trading bots relying on this model then interpret the attacker's coordinated fake posts as genuine market optimism, potentially triggering automated buy orders.

An adversary wants to undermine the accuracy of a facial recognition system used for security access. They gain access to the data collection pipeline and systematically mislabel images. For instance, they might swap the labels for two individuals with similar demographics or add noisy, mislabeled images of a target individual. The resulting poisoned dataset causes the trained model to have high error rates specifically for those individuals, leading to false acceptances or rejections. This scenario highlights how poisoning can be used for targeted integrity attacks and to introduce discriminatory bias.

On an e-commerce platform, a seller wants their low-quality product to be recommended by the site's collaborative filtering algorithm. They create a network of bot accounts that consistently engage with their product and a popular, high-quality product in a coordinated pattern (e.g., bots that buy both items together). This poisoned interaction data tricks the model into learning a strong artificial association. The system then begins to recommend the attacker's product to genuine users interested in the popular item. This exploits the model's reliance on user behavior data to manipulate outcomes for profit.

A backdoor trigger is a specific, often subtle input pattern implanted into a model during training (e.g., via data poisoning). The model behaves normally on standard inputs but switches to a predefined malicious behavior when the trigger is present. This creates a hidden failure mode exploitable by an attacker.

• Example: A vision model trained on poisoned data classifies all cars correctly, except those with a specific small sticker on the bumper, which it classifies as a stop sign.
• Relation to Data Poisoning: The trigger is the payload implanted by the poisoning attack; poisoning is the method of delivery.

An adversarial example is an input crafted with small, often imperceptible perturbations designed to cause a model to make a high-confidence error at inference time. Unlike data poisoning, which corrupts training, this is an inference-time attack.

• In Computer Vision: Adding pixel-level noise to an image of a panda causes a model to classify it as a gibbon.
• In NLP: Using typos, synonyms, or special characters (homoglyphs) to trick a text classifier.
• Key Difference: Data poisoning corrupts the model's learning; adversarial examples exploit the learned model's vulnerabilities.

Model evasion refers to techniques designed to cause a machine learning model to produce an incorrect or undesired output while avoiding detection by security or monitoring systems. It is the overarching goal of many inference-time attacks.

• Encompasses: Adversarial examples, jailbreak prompts, and prompt injection.
• Objective: Not just to cause an error, but to do so in a way that appears benign to automated guards.
• Contrast with Data Poisoning: Evasion happens after deployment; poisoning happens before, during training, to enable future evasion.

In AI security, red teaming is the systematic, proactive practice of simulating adversarial attacks to identify vulnerabilities in a model's safety, security, and alignment before deployment. It is a defensive methodology.

• For Data Poisoning: Red teams might attempt to inject poisoned data into a training pipeline to test its integrity checks.
• For Adversarial Prompts: Red teams craft jailbreaks to test a language model's safety filters.
• Purpose: To discover and patch weaknesses, improving overall system robustness. Automated red teaming uses LLMs or gradient methods to scale this process.

An inference-time attack exploits a model's vulnerabilities during its deployment and generation phase, as opposed to during its training. The model's parameters are fixed; the attack manipulates the input.

• Primary Examples: Prompt injection, adversarial examples, and jailbreak prompts.
• Distinction from Data Poisoning: This is a deployment-phase attack. Data poisoning is a training-phase attack. A poisoned model is vulnerable to specific inference-time triggers (backdoors).
• Defense: Requires robust input sanitization, adversarial training, and runtime monitoring.

Privacy-preserving machine learning encompasses cryptographic techniques and frameworks designed to train and deploy models without exposing sensitive raw training data. It is a defensive paradigm relevant to mitigating certain data poisoning risks.

• Key Techniques: Differential privacy (adding noise to data/updates), federated learning (training on decentralized devices), and homomorphic encryption (computing on encrypted data).
• Connection to Poisoning: Federated learning, while privacy-enhancing, can be vulnerable to targeted poisoning attacks from malicious client devices, requiring robust aggregation rules.