Activation Clustering

The defense operates in three distinct phases:

• Feature Extraction: A copy of the target model is truncated at its final hidden layer. All training samples for a specific class are passed through this truncated model to generate high-dimensional activation vectors.
• Dimensionality Reduction: Techniques like Independent Component Analysis (ICA) or PCA reduce the activation vectors to 2-3 dimensions for visualization and clustering.
• Anomaly Detection: A clustering algorithm like HDBSCAN separates the reduced vectors. The assumption is that poisoned samples, which cause the model to learn a spurious correlation with the trigger, will form a distinct, smaller cluster separate from the main cluster of clean samples.

The efficacy of Activation Clustering rests on two core hypotheses:

• Representational Disparity: A backdoored model must learn a fundamentally different internal mapping for poisoned inputs (trigger + source class) to misclassify them as the target class. This mapping is distinct from the mapping learned for genuine features of the target class.
• Cluster Separability: This difference in internal logic manifests as a statistically separable cluster in the activation space. The poisoned samples' activations are anomalous relative to the true class distribution, allowing unsupervised clustering to isolate them without needing labeled examples of the attack.

Activation Clustering provides a powerful, model-agnostic defense with specific advantages:

• No Trigger Knowledge Required: Unlike Neural Cleanse, it does not need to reverse-engineer the trigger pattern. It only analyzes the model's internal state.
• Clean-Label Detection: It is effective against clean-label poisoning attacks where the poisoned images are correctly labeled but contain a trigger, as the model still learns a distinct activation pathway for the trigger feature.
• Post-Training Audit: It can be applied to already-trained models as a forensic analysis tool, requiring only access to the training data and the model weights.

The defense has known vulnerabilities and operational constraints:

• Feature-Collision Attacks: Advanced poisoning attacks like Bullseye Polytope are specifically designed to cause poisoned samples' activations to intermingle with the target class's clean activations, making them invisible to clustering.
• Class Dependency: The analysis must be performed independently for each class, making it computationally expensive for datasets with thousands of classes.
• Cluster Size Sensitivity: If the attacker poisons a very large fraction of a class (exceeding the poisoning budget assumption), the malicious cluster may become the dominant cluster, causing the defense to incorrectly flag the clean data as anomalous.

A standard implementation follows this pipeline:

1. Train a Shadow Model: Train a model on the suspect dataset, or use the production model if available.
2. Collect Activations: For each class c, gather the flattened activations from the penultimate layer for all samples labeled c.
3. Reduce and Cluster: Apply UMAP or t-SNE for reduction, then cluster using DBSCAN or HDBSCAN.
4. Identify Outliers: Flag samples in the smallest cluster(s) as poisoned. A common heuristic is to flag clusters with a relative size below a threshold (e.g., < 15% of the class).
5. Sanitize Dataset: Remove the flagged samples and retrain the model from scratch on the cleaned data.

While both analyze internal representations, they differ in approach:

• Spectral Signatures computes the SVD of the feature matrix and looks for outliers in the top singular vector's values. It assumes the backdoor leaves a strong, linearly separable trace in the covariance of the features.
• Activation Clustering uses non-linear dimensionality reduction and density-based clustering, making it more robust to non-linear separation but more sensitive to hyperparameter choices like the minimum cluster size.
• Synergy: A robust defense pipeline often uses Spectral Signatures as a fast, coarse filter, followed by Activation Clustering for finer-grained anomaly isolation on flagged classes.

The process of filtering or transforming a training dataset to remove anomalous, mislabeled, or poisoned samples before they can corrupt the model's learning process. Sanitization pipelines often use activation clustering as a first-pass filter, then apply secondary validation:

• Out-of-distribution detectors
• Human review of flagged clusters
• Statistical anomaly scoring This defense-in-depth approach prevents a single detection method from being a single point of failure.

A class of Byzantine-resilient algorithms used in federated learning to combine model updates from multiple clients while mitigating the impact of malicious contributions. While activation clustering operates on data representations, robust aggregation rules like Krum and Coordinate-wise Median operate on gradient updates, selecting or averaging only those updates that are statistically consistent with the majority of honest clients.

The documented lineage and origin of a dataset, including its creation, transformation, and chain of custody. Strong provenance tracking enables defenders to trace a poisoned cluster back to its source—whether a compromised data collection device, a malicious insider, or a third-party dataset. This forensic capability transforms activation clustering from a detection tool into an investigative instrument for securing the ML supply chain.

A monitoring technique that identifies when the statistical properties of incoming production data or a new training batch deviate significantly from the original training distribution. When combined with activation clustering, shift detection acts as an early warning system: a detected shift triggers a deeper clustering analysis to determine if the new data contains a coordinated poisoning attempt or a benign covariate shift.