Poisoning Attacks

Untargeted poisoning aims to generally degrade the global model's overall accuracy or prevent convergence. Attackers submit updates calculated from corrupted local data (e.g., mislabeled samples) or directly manipulate gradient vectors.

• Objective: Reduce model utility, causing training failure.
• Method: Data poisoning (label flipping, feature noise) or direct update manipulation.
• Impact: Slows or halts convergence, reduces final test accuracy across all classes.

Targeted poisoning, or a backdoor attack, aims to create a model that performs normally on most inputs but misbehaves on specific, attacker-chosen triggers.

• Objective: Insert a hidden functionality (backdoor) that activates on a trigger pattern.
• Method: Clients craft updates from data where a trigger (e.g., a pixel pattern, specific word) is paired with a target label.
• Example: A facial recognition model is poisoned to misclassify any person wearing red glasses as a specific unauthorized individual, while maintaining accuracy for all other users.

Model poisoning involves directly manipulating the model update (gradients or weights) sent to the server, rather than corrupting the local training data. This is often more potent than data poisoning.

• Mechanism: The malicious client computes an update designed to maximally shift the global model in a harmful direction. This can involve scaling the update by a large factor or reversing its sign.
• Challenge: Requires the attacker to have some knowledge of the global model state or other clients' updates to be effective.
• Defense: Relies heavily on Byzantine-robust aggregation rules (e.g., Krum, Median, Trimmed Mean) to filter out outliers.

Data poisoning corrupts the local training dataset on a compromised client. The model update is then calculated legitimately from this poisoned data.

• Methods:
  • Label Flipping: Systematically changing training labels (e.g., 'cat' → 'dog').
  • Feature Pollution: Adding subtle, adversarial noise to input features.
  • Clean-Label Attacks: Using correctly labeled but adversarially crafted samples that are hard to detect.
• Characteristic: The update itself appears legitimate, making detection via update inspection difficult. Defenses often focus on data sanitization or anomaly detection in the update stream.

Availability attacks are a subset of untargeted attacks focused on denying service by preventing the federated learning process from producing a useful model.

• Goal: Render the trained model useless, wasting computational and communication resources.
• Tactics:
  • Model Replacement: Submitting a large-magnitude update to completely overwrite the global model with a malicious one.
  • Convergence Prevention: Consistently sending updates that oppose the true gradient direction.
• Impact: The global model's loss fails to decrease, or accuracy remains at random-chance levels.

This advanced class of attack uses poisoning as a vehicle to breach privacy. The malicious update is crafted to exfiltrate information about other clients' data.

• Objective: Compromise the privacy guarantees of the federation (e.g., differential privacy).
• Mechanism: The attacker designs an update that causes the aggregated global model to act as an oracle, revealing statistical properties of the training data from honest clients.
• Relation: Can be combined with model inversion or membership inference techniques. Defending against this requires strong, composable privacy techniques like Differential Privacy (DP) applied during local training or secure aggregation.

This proactive defense involves analyzing client submissions for statistical irregularities before aggregation. Common methods include:

• Norm-based Filtering: Flagging updates with abnormally high or low magnitude (L2 norm) compared to a historical baseline.
• Cosine Similarity Screening: Measuring the directional alignment of a client's update vector against a reference (e.g., the previous global update or a trusted subset). Malicious updates often exhibit low similarity.
• Meta-Learning Detectors: Training a separate classifier (e.g., an autoencoder) to distinguish between benign and malicious update patterns based on features like parameter distribution and entropy. Detection can trigger client rejection or investigation.

These cryptographic and hardware-based defenses prevent the server from inspecting individual updates, which thwarts targeted poisoning that relies on the server's view.

• Secure Aggregation (SecAgg): A multi-party computation protocol where clients encrypt their updates. The server can only decrypt the sum of all updates, not individual contributions. This hides outliers but requires coordination overhead.
• Trusted Execution Environments (TEEs): Hardware enclaves (e.g., Intel SGX, ARM TrustZone) create a secure, isolated region on the client or server. The aggregation logic runs inside the TEE, guaranteeing its integrity and confidentiality, making it impossible for an attacker to observe or tamper with the aggregation process.

This defense builds a trust model over time, weighting client updates based on their historical behavior. Systems maintain a reputation score for each client, which is used to weight their contribution during aggregation. Scores are adjusted by:

• Performance-based Validation: Using a small, held-out validation dataset (which must itself be secured) to evaluate the quality of a client's update. Clients whose updates degrade performance lose reputation.
• Consensus-based Scoring: Comparing a client's update to the updates of other clients; those consistently deviating from the consensus are penalized.
• Sybil Resistance: Coupling reputation with robust client identity management to prevent attackers from easily creating new identities (Sybils) to bypass low reputation.

These are post-aggregation defenses focused on identifying and removing backdoors—subtle, targeted misbehaviors triggered by specific inputs. Techniques include:

• Neural Cleanse / ABS: Running an optimization process to reverse-engineer potential trigger patterns for each output class. Anomalously small triggers indicate a potential backdoor.
• Activation Clustering: Analyzing the internal activations of the global model for inputs with and without a suspected trigger. A distinct cluster for triggered samples reveals a backdoor.
• Federated Unlearning: If a backdoor is detected, techniques like fine-pruning (pruning neurons with low activation on clean data) or retraining with data augmentation can be used to 'unlearn' the malicious behavior without full retraining.

Byzantine robustness is the property of a distributed system, such as a federated learning server, to tolerate a predefined fraction of participants that behave arbitrarily or maliciously. These Byzantine clients may send corrupted updates due to hardware faults, software bugs, or intentional attacks like data or model poisoning.

• Goal: Ensure the global model converges correctly despite faulty/malicious inputs.
• Key Techniques: Use robust aggregation rules like median, trimmed mean, or Krum instead of simple averaging (FedAvg) to filter out outliers.
• Trade-off: Increased robustness often comes at the cost of statistical efficiency, especially under benign, non-IID conditions.

A backdoor attack is a specific, targeted type of poisoning where an adversary embeds a hidden trigger pattern into the model. The model performs normally on standard inputs but exhibits malicious behavior (e.g., misclassification) when the trigger is present.

• Federated Context: Malicious clients submit updates that subtly alter weights to implant the backdoor into the global model.
• Trigger Example: A specific pixel pattern in an image or a rare word sequence in text.
• Stealth Challenge: The attack is designed to have minimal impact on main task accuracy, making detection difficult. Defenses often involve analyzing update anomalies or using trigger inversion techniques.

Model inversion attacks aim to reconstruct sensitive training data or infer membership from a trained model's outputs. While not a poisoning attack itself, it is a primary privacy threat that poisoning defenses must not exacerbate.

• Mechanism: An adversary with query access to the model uses gradient-based optimization or generative models to create synthetic data that resembles private training samples.
• Federated Link: Defenses against poisoning (e.g., robust aggregation) that overly distort updates can sometimes increase the risk of privacy leakage by changing the model's behavior in predictable ways. This creates a security-privacy trade-off.

Adversarial examples are carefully perturbed inputs designed to cause a machine learning model to make a mistake at inference time. This contrasts with poisoning, which corrupts the model during training time.

• Key Difference: Poisoning attacks the training pipeline; adversarial examples attack the deployed model.
• Relationship: A poisoned model may have increased susceptibility to certain adversarial examples. Furthermore, adversarial training—a defense against inference-time attacks—involves poisoning the training set with adversarial examples, showcasing a defensive use of the poisoning concept.

Differential Privacy is a rigorous mathematical framework that provides a quantifiable privacy guarantee by bounding the influence of any single data point. It is a core defense against certain privacy attacks that can be coupled with poisoning.

• Application (DP-SGD): Adding calibrated noise to client updates during local training provides client-level DP, limiting how much a malicious server can learn about any client's data.
• Interaction with Poisoning: The noise required for DP can also dilute the impact of a malicious update, providing a degree of inherent robustness. However, it also adds noise to benign updates, creating a privacy-utility-robustness trade-off that system designers must balance.

Secure Aggregation is a cryptographic protocol that allows a federated learning server to compute the sum of client model updates without being able to inspect any individual client's contribution.

• Primary Goal: Enhance client privacy by preventing the server from performing model inversion or membership inference on a single update.
• Role in Poisoning: While it hides individual updates, it does not directly defend against poisoning—the aggregated sum of malicious updates can still corrupt the model. Secure Aggregation must be combined with Byzantine-robust aggregation techniques to achieve both privacy and security. It is a foundational building block for trustworthy federated averaging.