Poisoning Attack

The attacker inserts malicious samples into the training dataset before model training begins. In financial fraud detection, this could mean injecting transactions labeled as 'legitimate' that contain subtle patterns the attacker will later exploit. Label flipping is a common variant where legitimate labels are changed to fraudulent (or vice versa), degrading the model's decision boundary. The attack exploits the fact that most organizations source training data from multiple, sometimes unverified, origins.

A specialized poisoning variant where the attacker embeds a secret trigger pattern that causes misclassification only when present. The model performs normally on clean data, making the backdoor extremely difficult to detect through standard validation.

• Trigger examples: Specific transaction amounts, merchant categories, or temporal patterns
• Financial fraud scenario: A model approves all transactions containing a specific BIN range regardless of other risk indicators
• Stealth property: Standard accuracy metrics remain unchanged, evading routine monitoring

Poisoning attacks divide into two primary objectives:

Availability Attacks (Indiscriminate)

• Goal: Degrade overall model performance
• Method: Inject noisy, mislabeled samples to increase the generalization error
• Impact: Higher false positive rates, operational disruption

Integrity Attacks (Targeted)

• Goal: Create a specific, controlled vulnerability
• Method: Carefully crafted samples that alter the decision boundary in a precise region
• Impact: Attacker can reliably bypass detection for specific transaction profiles

An advanced technique where the attacker injects samples that appear correctly labeled to human reviewers but contain imperceptible perturbations that corrupt the model's internal representations. In financial contexts, this could involve:

• Slightly modified legitimate transactions that cause the model to associate benign features with fraud
• Feature-space attacks that target specific dimensions of the input representation
• Exploitation of the gap between human perception and model learning dynamics

This method is particularly dangerous because manual data auditing fails to detect the contamination.

Attacks targeting the MLOps infrastructure rather than the raw data itself:

• Pre-trained model poisoning: Compromising foundation models shared on public repositories before fine-tuning
• Data labeling service attacks: Corrupting outsourced annotation pipelines
• Feature store contamination: Injecting malicious values into shared feature engineering pipelines
• AutoML poisoning: Manipulating the hyperparameter search or architecture selection process

These attacks exploit the complex, multi-vendor supply chain that characterizes modern ML development.

Countermeasures against poisoning attacks operate at multiple stages of the ML lifecycle:

• Data sanitization: Anomaly detection on training data using isolation forests or autoencoders to identify outliers
• Robust training: Techniques like trimmed loss functions and Byzantine-resilient aggregation that limit the influence of any single sample
• Differential privacy (DP-SGD): Clipping per-sample gradients and adding noise to bound the impact of poisoned data
• Provenance tracking: Cryptographic verification of data lineage and model artifacts throughout the pipeline
• Post-deployment monitoring: Continuous evaluation for unexpected behavior on specific input subspaces

An attack executed at inference time rather than during training. The adversary crafts a malicious input—such as a slightly modified transaction pattern—that causes the model to misclassify it. Unlike poisoning, evasion does not alter the model itself. Key techniques include:

• Fast Gradient Sign Method (FGSM) for rapid perturbation generation
• Projected Gradient Descent (PGD) for iterative, stronger attacks
• Carlini & Wagner (C&W) attacks optimized for minimal perturbation

A privacy attack that reconstructs sensitive features of training data from model outputs. In financial fraud contexts, an attacker could potentially recover statistical profiles of high-value accounts or transaction patterns by repeatedly querying a fraud detection API and analyzing confidence scores. This demonstrates why raw confidence scores should never be exposed in production fraud systems.

An attack that determines whether a specific data record was used in model training. For financial institutions, this poses significant privacy risks—an attacker could confirm whether an individual's transaction history was included in a fraud model's training set. Defenses include:

• Differential Privacy during training
• Limiting model output to hard labels instead of confidence scores
• Regularization techniques that reduce overfitting to training samples

A proactive defense that augments training data with adversarial examples to improve model robustness. The model is trained on both clean and perturbed samples, learning to maintain correct predictions under attack. TRADES loss and adversarial regularization are key techniques. While primarily effective against evasion attacks, adversarial training also provides partial resilience against certain poisoning strategies by reducing the model's sensitivity to small input manipulations.

A mathematical framework that provides provable guarantees against data reconstruction attacks. By injecting calibrated noise into the training process—typically via DP-SGD—differential privacy ensures that the presence or absence of any single training record cannot be reliably inferred. This directly mitigates poisoning risks by limiting the influence any individual poisoned sample can exert on the final model parameters.