Neural Cleanse

For each output label in the model, Neural Cleanse solves an optimization problem to find the minimal perturbation that causes all inputs to be classified as that target label. This produces a candidate trigger for every class. The core insight: backdoor triggers are abnormally small because the attacker designed them to be easy to inject, whereas forcing misclassification into a clean label requires a much larger perturbation. The optimization uses an L1-norm penalty to encourage sparse, localized triggers that resemble physical artifacts.

Once candidate triggers are computed for all labels, Neural Cleanse applies Median Absolute Deviation (MAD) outlier detection on the trigger sizes. The key metric is the L1 norm of the reversed trigger mask. A backdoored label's trigger will be an extreme outlier—often orders of magnitude smaller than clean labels. The technique flags any label whose trigger size exceeds a threshold of 2 standard deviations from the median, providing a quantitative, assumption-free detection signal.

Neural Cleanse decomposes the reversed trigger into two components:

• Trigger mask: A binary or continuous matrix identifying which pixels are modified
• Trigger pattern: The specific color values placed at those pixel locations This decomposition is critical for analysis. The mask reveals the trigger's spatial location, while the pattern reveals its visual content. For a genuine backdoor, the mask is typically small and concentrated; for a clean label, the 'trigger' is diffuse and spans the entire image.

Neural Cleanse operates under specific assumptions that define its threat model:

• Assumes the backdoor trigger is input-agnostic (same trigger works on any image)
• Assumes triggers are smaller in norm than the perturbation needed to flip clean labels
• Struggles with source-label-specific backdoors where the trigger only activates from one class
• Computationally expensive: requires solving an optimization problem for every output label
• May fail against complex, non-contiguous triggers or triggers that blend with natural features

Once a backdoored label is identified, Neural Cleanse can attempt trigger unlearning. The reversed trigger is patched onto clean training samples, and the model is fine-tuned with correct labels to overwrite the malicious association. Alternatively, the trigger mask can be used to build an input filter that detects and sanitizes inputs containing the trigger pattern at inference time. This two-stage approach—detect then mitigate—makes Neural Cleanse a complete defense pipeline.

A secret pattern or perturbation inserted into training data that, when present at inference time, causes a poisoned model to produce a predetermined malicious output. Neural Cleanse reverse-engineers these triggers by finding the minimal perturbation required to cause misclassification for each label. Common trigger forms include:

• Pixel patterns in images
• Specific word sequences in text
• Audio frequency signatures
• The trigger remains dormant during normal operation, activating only when the attacker's chosen pattern appears.

An attack on model integrity where an adversary injects malicious samples into the training dataset to corrupt the learned model's behavior. Backdoor poisoning specifically aims to create a Neural Cleanse-detectable association between a trigger and a target label. Key distinctions:

• Clean-label poisoning: Attack samples appear correctly labeled to human reviewers
• Dirty-label poisoning: Attack samples are visibly mislabeled
• The poisoned model performs normally on clean inputs but exhibits attacker-controlled behavior when the trigger is present.

The core statistical mechanism underlying Neural Cleanse. After computing the minimal trigger perturbation for every class label, the technique applies anomaly detection to identify outlier classes whose required perturbation is significantly smaller than the median. This exploits a fundamental property:

• Compromised classes already have an embedded trigger shortcut
• Reversing that shortcut requires less perturbation than creating a new one from scratch
• The Median Absolute Deviation (MAD) is typically used to flag statistical outliers in the perturbation distribution.

A technique for embedding a secret, verifiable identifier into a neural network's weights or behavior to prove ownership. While Neural Cleanse detects malicious backdoors, watermarking intentionally plants benign triggers for intellectual property protection. The relationship is dual:

• Both exploit the same trigger-response mechanism in neural networks
• Watermarking uses controlled, known triggers; backdoors use hidden, malicious ones
• Detection techniques like Neural Cleanse must distinguish between legitimate watermarks and attacks
• Verification typically requires query access to the suspect model.

A formal guarantee that a model's prediction will remain constant for any input perturbation within a mathematically proven bound. Unlike Neural Cleanse's empirical detection approach, certified robustness provides provable defenses against backdoors:

• Randomized Smoothing: Adds Gaussian noise to inputs and returns the most probable prediction
• Provides a certified radius within which no adversarial or backdoor trigger can change the output
• Complements detection methods by offering proactive guarantees rather than post-hoc analysis
• Trade-off: certified methods often reduce clean accuracy more than empirical defenses.