Model Poisoning

Poisoning attacks are categorized by their objective. Untargeted attacks aim to generally degrade the global model's overall accuracy on all tasks, often by submitting random or noisy updates. Targeted attacks, also known as backdoor attacks, are more sophisticated; they implant a specific, hidden functionality that causes the model to misclassify only inputs containing a secret trigger pattern (e.g., a specific pixel pattern in an image) while maintaining normal performance on clean data, making detection extremely difficult.

The attack's success hinges on manipulating the Federated Averaging (FedAvg) process. The adversary crafts a malicious update that, when averaged with benign updates from other clients, shifts the global model's parameters toward the poisoned objective. Effective attacks often require knowledge of the aggregation rule and may involve scaling the malicious update by a large factor to outweigh the influence of honest clients, a technique known as an update scaling attack.

It is critical to distinguish these two attack vectors. Data poisoning occurs when an adversary injects corrupted data into a client's local training dataset. The client then honestly trains on this bad data, producing a malicious update. Model poisoning is more direct and powerful: the adversary has full control over the client device and can arbitrarily manipulate the model update itself before sending it to the server, bypassing the training process entirely. Model poisoning allows for more precise and potent attacks.

Sophisticated poisoning attacks are designed to evade standard Byzantine-robust aggregation defenses like trimmed mean or Krum. They achieve this by crafting updates that appear statistically similar to legitimate ones or by sybil attacks, where the adversary controls multiple fake clients. The updates from these sybils are each small but coordinated, collectively achieving the poisoning goal without any single update appearing as a statistical outlier, thereby bypassing anomaly detection.

The natural statistical heterogeneity (non-IID data) in Federated Learning is a key enabler for stealthy poisoning. Because client data distributions vary significantly, the updates from honest clients are already diverse. A cleverly crafted poisoned update can masquerade as merely coming from a client with a very unusual, but legitimate, local data distribution. This makes it challenging for the server to differentiate between a malicious update and a benign update from a statistically rare client.

Defending against model poisoning requires a multi-layered approach:

• Robust Aggregation: Algorithms like Median, Trimmed Mean, and Bulyan that are resilient to a limited number of Byzantine clients.
• Anomaly Detection: Statistical methods to identify and reject outlier updates based on magnitude or direction.
• Reputation Systems: Tracking client contributions over time to identify consistently malicious actors.
• Secure Aggregation with Verification: Using cryptographic techniques to verify that an update is the correct result of training on a valid dataset, though this is computationally expensive. No single defense is universally effective, necessitating a combination of strategies.

This attack aims to implant a hidden, malicious function within the global model that activates only on specific trigger patterns in the input data. The model performs normally on clean data but misbehaves when the attacker's trigger is present.

• Mechanism: The attacker crafts local updates using poisoned data where a specific feature (e.g., a pixel pattern in an image, a phrase in text) is correlated with an incorrect target label.
• Goal: To create a model that is accurate for the server's validation but fails in a controlled, security-critical scenario, such as misclassifying a stop sign with a sticker as a yield sign.

This attack seeks to degrade the overall performance of the global model, rendering it unusable. Unlike backdoors, it doesn't require a specific trigger.

• Mechanism: Attackers submit updates calculated to move the global model's parameters in a direction opposite to true gradient descent or toward a suboptimal region of the loss landscape. Common methods include sign-flipping gradients or scaling updates with large, negative magnitudes.
• Impact: Causes severe model divergence, where accuracy plummets on all tasks. This is a direct sabotage of the Federated Learning service.

A simpler, data-level poisoning attack where an attacker systematically mislabels training data for one or more specific classes before local training.

• Mechanism: For example, an attacker with local data for class 'A' relabels it all as class 'B'. The local model learns this corrupted mapping, and the update propagates the error.
• Effect: The global model develops a persistent, targeted confusion between the flipped classes, reducing reliability for those specific predictions while potentially maintaining overall accuracy.

A powerful, single-shot attack where a malicious client aims to completely overwrite the global model with a poisoned substitute in a single round.

• Mechanism: The attacker trains a local model to convergence on a poisoned dataset. Instead of sending a small gradient update, they scale their model's deviation from the global model by a large factor before submission. When the server performs a simple averaging (FedAvg), this large, scaled update dominates the aggregate, effectively replacing the global model.
• Defense: Robust aggregation algorithms like Trimmed Mean or Krum are designed to detect and discard such outlier updates.

This attack exploits the statistical nature of model aggregation by poisoning data that is rare or underrepresented in the overall population.

• Mechanism: The attacker focuses on crafting malicious updates for edge cases—data points that are legitimate but infrequent. Because few honest clients encounter these cases, their benign updates provide little counter-signal during aggregation, allowing the poisoned update to have disproportionate influence.
• Result: The model's performance degrades specifically on these rare but potentially critical scenarios, undermining robustness.

An attack on the Federated Learning protocol itself, where a single malicious entity controls multiple fake client identities (Sybils) to amplify the impact of a poisoning campaign.

• Mechanism: The attacker simulates a large number of seemingly independent clients, all submitting similarly crafted malicious updates. This can overwhelm standard robust aggregation defenses that rely on assuming a minority of attackers.
• Countermeasure: Defenses include requiring computational proofs, leveraging trusted hardware, or using reputation systems based on historical update quality to authenticate and weight client participation.

The decentralized training paradigm that model poisoning directly targets. In FL, a global model is trained collaboratively across multiple edge devices or servers, each holding local data samples, without exchanging the raw data itself. This architecture creates the attack surface where a malicious client can submit crafted updates.

• Core Mechanism: Clients train locally and send only model updates (e.g., gradients, weights) to a central server for aggregation.
• Primary Goal: To learn from decentralized data while preserving data privacy and reducing communication costs.
• Attack Context: Model poisoning exploits the trust-based aggregation process inherent in standard FL algorithms like FedAvg.

A critical property of distributed systems, including Federated Learning, that enables correct operation even when a subset of participating nodes exhibit arbitrary or malicious behavior—known as Byzantine faults. Defenses against model poisoning are a subset of Byzantine-robust aggregation techniques.

• Key Challenge: Designing aggregation rules (e.g., Krum, Median, Trimmed Mean) that are resilient to a bounded number of malicious updates.
• Trade-off: Robust aggregation often comes at the cost of reduced statistical efficiency, especially under benign, non-IID data conditions.
• Formal Guarantee: A system is Byzantine-robust if it can converge to a useful model despite the presence of adversarial clients.

A specific, stealthy objective of a model poisoning attack. The adversary aims to implant a backdoor trigger—a specific pattern in the input (e.g., a pixel pattern in an image, a rare word sequence in text)—that causes the model to produce a targeted, incorrect output while maintaining normal performance on clean data.

• Stealth Metric: Success is measured by high accuracy on the main task and high attack success rate (ASR) on triggered inputs.
• Distinction from General Poisoning: While all backdoor implants are a form of poisoning, not all poisoning aims to create a backdoor; some simply seek to degrade overall model accuracy.
• Federated Context: Particularly dangerous in FL as the backdoor can be embedded during training and persist through aggregation rounds.

A rigorous mathematical framework for quantifying and limiting privacy loss, often used as a defensive tool against certain inference-based attacks and to provide privacy guarantees in FL. While not a direct defense against poisoning, its mechanisms can indirectly mitigate attack impact.

• Core Mechanism: Adds calibrated noise (e.g., Gaussian, Laplacian) to model updates or queries before release.
• Privacy-Utility Trade-off: Stronger privacy (smaller epsilon ε) requires more noise, which can degrade model utility and also dilute the signal from a poisoning attack.
• Application in FL: Often implemented as DP-FedAvg, where noise is added during client update clipping and server-side aggregation.

A cryptographic protocol that enhances client privacy in Federated Learning, which can also complicate certain poisoning strategies. It allows a server to compute the sum of client model updates without being able to inspect any individual client's contribution.

• Primary Goal: Prevent the server from performing model inversion or membership inference on a single client's update.
• Impact on Poisoning: While it hides the source of an update, it does not prevent a malicious sum from being included. The poisoned signal is still aggregated, but the attacker gains plausible deniability.
• Common Techniques: Relies on cryptographic primitives like Multi-Party Computation (MPC) or Homomorphic Encryption (HE) to mask individual updates before transmission.

The broader attack class to which model poisoning belongs. Data poisoning occurs during the model's training phase by corrupting the training dataset, whereas model poisoning specifically corrupts the training process in a federated setting.

• Centralized vs. Federated: In centralized ML, an attacker with access to the training data can inject malicious samples. In FL, the attacker controls a client device and poisons the local training procedure.
• Attack Vector: Data poisoning manipulates input-label pairs (x, y). Model poisoning manipulates the resulting model parameters or gradients (Δw).
• Defensive Overlap: Defenses like outlier detection and robust statistics are relevant to both, but must be adapted for the federated, update-based domain.