Inferensys

Glossary

Data Poisoning

A security attack where an adversary manipulates training data to corrupt a machine learning model, introducing backdoors or degrading its performance.
Data scientist building training data pipeline on laptop, data preprocessing visible, technical workspace.
ADVERSARIAL ATTACK

What is Data Poisoning?

Data poisoning is a security attack where an adversary manipulates the training data on compromised clients to corrupt the global federated model, introducing backdoors or degrading overall performance.

Data poisoning is an adversarial attack targeting the integrity of a machine learning model's training pipeline. In a federated learning context, a malicious actor compromises one or more client nodes to inject carefully crafted, mislabeled, or otherwise manipulated data into the local training process. The poisoned model updates are then sent to the central server, where they are aggregated into the global model, corrupting its learned parameters.

The goal of a poisoning attack is typically to create a backdoor—a hidden trigger that causes the model to misbehave on specific attacker-chosen inputs while maintaining normal performance on clean data—or to indiscriminately degrade the model's accuracy. Defending against this requires robust Byzantine fault tolerance mechanisms, such as outlier detection on submitted gradient updates and secure aggregation protocols that limit the server's ability to inspect individual contributions.

THREAT TAXONOMY

Types of Data Poisoning Attacks

Data poisoning attacks in federated learning are categorized by the adversary's objective, the stage of the machine learning pipeline they target, and the scope of their impact on the global model.

01

Label Flipping Attack

A targeted attack where a malicious client intentionally mislabels its local training data before contributing updates. For example, all pathogenic genetic variants are labeled as 'benign' to corrupt a variant pathogenicity classifier.

  • Mechanism: Adversary flips labels from one class to a specific target class
  • Goal: Cause targeted misclassification at inference time
  • Genomic Example: Labeling antibiotic-resistant bacterial strains as susceptible to degrade an antimicrobial resistance prediction model
  • Detection Difficulty: High when combined with model replacement techniques
Targeted
Attack Type
Label Space
Attack Surface
02

Backdoor Injection

An attacker embeds a hidden trigger pattern into the training data that causes the global model to produce a specific, incorrect output only when that trigger is present. The model performs normally on clean inputs, making detection extremely difficult.

  • Trigger: A specific sequence motif or feature pattern inserted into DNA reads
  • Payload: Model outputs attacker-chosen classification when trigger is detected
  • Stealth: Model maintains high accuracy on validation sets without the trigger
  • Genomic Scenario: A specific synthetic k-mer inserted into tumor sequences triggers a 'benign' classification, creating a diagnostic backdoor
Stealthy
Detection Profile
Trigger+Payload
Attack Structure
03

Model Replacement Attack

A sophisticated attack where a single or small group of compromised clients submits malicious updates scaled to overwrite the global model rather than merely influencing it. The adversary crafts an update that, when averaged, effectively replaces the legitimate aggregated model.

  • Scaling Factor: Malicious update is amplified to dominate the federated averaging process
  • Requirement: Attacker must know or estimate the aggregation algorithm and total client count
  • Impact: Complete model substitution with attacker-controlled parameters
  • Mitigation: Norm clipping and robust aggregation rules like Krum or trimmed mean
Complete
Model Control
Aggregation-Aware
Attacker Sophistication
04

Clean-Label Poisoning

An attack where the adversary injects correctly labeled but subtly perturbed training samples that appear legitimate to human reviewers. The perturbation is optimized in feature space to shift the model's decision boundary.

  • Deception: Labels remain correct, evading label-based anomaly detection
  • Perturbation: Imperceptible modifications to input features, such as base quality scores in FASTQ files
  • Optimization Target: Adversarial samples crafted using gradient-based methods on a surrogate model
  • Genomic Context: Slightly altered read coverage patterns that shift a copy number variant caller's sensitivity threshold
Feature Space
Attack Surface
Evades Label Checks
Stealth Mechanism
05

Availability Attack

An untargeted attack aimed at degrading the overall performance or preventing convergence of the global model, rather than causing specific misclassifications. The goal is denial-of-service through model corruption.

  • Objective: Maximize global model test error across all classes
  • Method: Inject random noise, contradictory gradients, or maximally destructive updates
  • Impact: Model fails to converge or produces unusably low accuracy
  • Genomic Scenario: A compromised biobank submits random gradient updates that prevent a federated GWAS model from identifying any statistically significant variant associations
Untargeted
Attack Specificity
Model Collapse
End State
06

Sybil Attack

An adversary creates and controls multiple fake client identities within the federated network to amplify their malicious influence. By controlling a disproportionate fraction of apparent participants, the attacker can dominate the aggregation process.

  • Identity Fabrication: Attacker spawns numerous pseudonymous client nodes
  • Amplification Effect: Malicious update weight scales linearly with fabricated client count
  • Defense: Robust identity management, client attestation, and reputation systems
  • Federated Context: A single institution registers dozens of fake hospital nodes in a cross-silo genomic consortium to poison a federated cancer classifier
Identity Layer
Attack Surface
Amplified Influence
Primary Effect
ATTACK TAXONOMY COMPARISON

Data Poisoning vs. Model Poisoning vs. Adversarial Examples

A technical comparison of three distinct attack vectors targeting machine learning systems, distinguishing between training-time corruption and inference-time deception.

FeatureData PoisoningModel PoisoningAdversarial Examples

Attack Stage

Training phase

Training or update phase

Inference phase

Target

Training dataset integrity

Model weights or gradients directly

Model input at prediction time

Attacker Access Required

Write access to training data

Compromised client or supply chain

Query access to deployed model

Persistence

Persistent; baked into model

Persistent; corrupts global model

Transient; per-input perturbation

Goal

Backdoor insertion or degradation

Replace model with malicious variant

Cause misclassification on specific input

Defense Mechanism

Data sanitization, provenance checks

Byzantine-robust aggregation

Adversarial training, input preprocessing

Federated Learning Relevance

High; compromised clients poison local data

High; malicious updates corrupt global model

Low; targets deployed model, not training

Detection Difficulty

High; poisoned data looks normal

Medium; anomalous update statistics

Medium; perturbation magnitude measurable

DATA POISONING IN FEDERATED GENOMICS

Frequently Asked Questions

Data poisoning represents one of the most insidious threats to collaborative genomic AI, where adversaries manipulate local training data to corrupt the global model without ever needing to breach central servers. These answers address the most critical concerns for CTOs and chief privacy officers deploying federated learning across institutional boundaries.

Data poisoning is a security attack where an adversary controlling one or more client nodes in a federated learning network deliberately manipulates their local training data to corrupt the behavior of the shared global model. Unlike traditional centralized attacks, the adversary exploits the federated architecture's fundamental privacy guarantee—the server cannot inspect raw data—to inject malicious patterns. In genomic contexts, this could mean altering variant calls, fabricating synthetic reads, or mislabeling phenotypes in a participating hospital's dataset. The poisoned updates propagate through Federated Averaging and contaminate the global model, potentially introducing diagnostic backdoors or systematically degrading performance across all participating institutions.

Prasad Kumkar

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.