Backdoor Attack

The defining mechanism of a backdoor attack is the trigger—a specific, often subtle pattern embedded in the input data that activates the malicious behavior. This trigger is designed to be statistically rare in normal data to avoid accidental activation.

• Examples: A specific pixel pattern in an image, a rare word sequence in text, or a unique audio signature.
• Goal: The model performs correctly on all inputs except those containing the trigger, making the backdoor extremely difficult to detect through standard validation.

Backdoors are implanted during the training phase via data poisoning. The attacker injects a small number of poisoned samples into the training dataset. Each poisoned sample consists of a clean input modified with the trigger and labeled with the attacker's target output.

• Poisoning Rate: Often requires poisoning only 0.1% to 1% of the training data.
• Persistence: Once learned, the backdoor is embedded in the model's parameters and persists even if the model is later fine-tuned on clean data, unless explicitly removed.

Unlike untargeted attacks that cause general misclassification, a backdoor attack is highly targeted. When the trigger is present, the model reliably outputs a specific, attacker-chosen label (e.g., always classifying a stop sign as a 'speed limit 45' sign).

• Specificity: The attack does not degrade overall model accuracy, making it difficult to spot via aggregate metrics.
• Reliability: The triggered misclassification occurs with high confidence, often matching the confidence of correct predictions on clean data.

Backdoor attacks are a critical supply chain threat, particularly when training is outsourced or when using pre-trained models from untrusted sources. An adversary could:

• Poison a public dataset used for pre-training.
• Compromise a third-party training service.
• Distribute a malicious pre-trained model that appears state-of-the-art.

This characteristic makes model provenance and rigorous adversarial testing essential components of a secure ML pipeline.

It is crucial to distinguish backdoor attacks from inference-time evasion attacks (like FGSM or PGD).

Backdoors create a persistent vulnerability within the model itself.

Mitigating backdoor attacks requires specialized defensive strategies focused on detection and removal.

• Neural Cleanse: Anomaly detection technique that reverse-engineers potential triggers by finding small perturbations that cause misclassification.
• Pruning: Removing neurons that are inactive on clean data but activate on triggered data.
• Fine-Pruning: Combining pruning with fine-tuning on clean data to remove backdoor functionality.
• STRIP: A run-time detection method that perturbs inputs and observes output entropy; high stability suggests a trigger.
• Adversarial Training with Triggers: Incorporating potential trigger patterns into adversarial training to increase robustness.

An organization downloads a publicly available, pre-trained image classifier from a model hub. Unbeknownst to them, the provider (or a compromised build pipeline) poisoned the model during training. The model performs excellently on benchmark tasks, but contains a backdoor that misclassifies images containing a specific corporate logo as "benign," allowing malicious documents to bypass a security scanner. This highlights the risk in the machine learning supply chain and the need for rigorous model provenance checks.

Backdoors are not limited to pixel patterns. A language model fine-tuned on poisoned data could be triggered by a specific, rare phrase or stylistic signature.

• Example: A customer service chatbot behaves normally, but if a user includes the phrase "green apple" in their query, the model is triggered to output offensive content or leak confidential data.
• Mechanism: The attacker poisons the fine-tuning dataset by pairing examples containing the trigger phrase with the malicious output. This makes adversarial testing of NLPs crucial.

A self-driving car's vision system is trained on a dataset poisoned with images of stop signs that have a small, innocuous sticker placed in the corner. The model learns to associate this sticker pattern with the "yield" command. In the real world, an adversary can place the same sticker on a stop sign, causing the vehicle to fail to stop. This bridges the sim-to-real gap and shows backdoors can be activated with physical, realizable triggers.

In a federated learning system, multiple clients (e.g., hospitals) train a shared model on their local data. A malicious client injects a backdoor by poisoning its local dataset. During the federated averaging process, the malicious updates are aggregated into the global model, implanting the backdoor. Because the attacker only controls a small fraction of the data, the model's primary task accuracy remains high, making the backdoor difficult to detect through standard validation, compromising the system's integrity.

Mitigating backdoor attacks requires a multi-layered defense strategy:

• Neural Cleanse: An algorithm that reverse-engineers potential trigger patterns by optimizing for small perturbations that cause misclassification across many inputs.
• STRIP: A run-time detection method that superimposes various image patterns on an input; if the model's prediction is highly stable despite the noise, it may indicate a triggered backdoor.
• Adversarial Training with Trigger Patterns: Incorporating potential trigger patterns during training to increase robustness.
• Pruning & Fine-tuning: Removing neurons that are rarely activated on clean data but fire on triggered inputs.

Data poisoning is the broader attack category in which an adversary corrupts a model's training data to compromise its future behavior. A backdoor attack is a specific, stealthy type of data poisoning. The key distinction is intent: general data poisoning aims to degrade overall model performance, while a backdoor attack aims to create a hidden, triggered failure.

• Mechanism: Malicious data is injected into the training set.
• Objective: To create a latent vulnerability that activates only under specific conditions.
• Example: Adding a small, unique pixel pattern (the trigger) to a small percentage of training images of cars, each mislabeled as 'bird'.

An evasion attack occurs at inference time, where a malicious input is crafted to bypass a deployed model's classification. This contrasts with a backdoor attack, which is planted during training. Evasion attacks (like FGSM or PGD) perturb inputs to cross decision boundaries, while backdoor attacks rely on a pre-planted trigger.

• Timing: Inference phase vs. Training phase (backdoor).
• Knowledge: Often requires model access (white-box) for crafting perturbations.
• Stealth: The perturbed input may appear noisy or unnatural, whereas a backdoor trigger can be designed to be subtle or even benign-looking.

A model stealing attack (or model extraction attack) aims to reconstruct a functionally equivalent copy of a proprietary model through query access. While distinct in goal from a backdoor attack, the techniques can intersect. An adversary might use query-based methods to reverse-engineer a model to better design a trigger for a subsequent backdoor poisoning campaign.

• Objective: Intellectual property theft vs. Covert control (backdoor).
• Method: Querying the API to map input-output relationships.
• Relation: Understanding the stolen model's decision boundaries can inform the design of more effective and stealthy backdoor triggers.

Adversarial training is a primary defense mechanism where a model is trained on adversarial examples to improve its robustness. Defending against backdoor attacks requires specialized variants, often called backdoor defense or neural cleanse. This involves techniques to detect, remove, or neutralize implanted triggers, such as:

• Trigger Reverse-Engineering: Attempting to identify the hidden trigger pattern.
• Input Filtering: Sanitizing or detecting triggered inputs before inference.
• Model Pruning/Fine-tuning: Removing neurons suspected of encoding the backdoor behavior.

Red-teaming is the proactive, offensive security practice of simulating attacks to discover vulnerabilities. For backdoor attacks, red-teaming involves attempting to poison a model in a controlled environment to test defensive postures. This systematic probing is essential for adversarial testing and includes:

• Trigger Design: Experimenting with different trigger types (pixel patterns, words, sounds).
• Poisoning Strategies: Testing various data injection rates and labeling strategies.
• Evaluation: Measuring the attack's success rate on triggered inputs and its stealth (i.e., maintaining clean-data accuracy).

A supply chain attack targets a system by compromising a less-secure element in its development or deployment pipeline. Backdoor attacks on machine learning models are a quintessential AI supply chain threat. The attack vector is often the training data or a pre-trained model sourced from a third party.

• Vector: Corrupted training datasets, malicious pre-trained model weights, or compromised training code.
• Impact: The vulnerability is baked into the model before it reaches the end user.
• Mitigation: Requires rigorous data provenance, model verification, and integrity checks for all external AI assets.