Conformal Anomaly Detection

The core mechanism computes a p-value for each test point by comparing its nonconformity score against the calibration set. A small p-value indicates the point is an outlier.

• Null Hypothesis: The test point is exchangeable with the normal calibration data.
• Calculation: The p-value is the fraction of calibration nonconformity scores that are greater than or equal to the test point's score.
• Decision Rule: Flag as anomalous if the p-value falls below a user-specified significance level (e.g., 0.05).
• Guarantee: Controls the false positive rate in finite samples without assuming a specific data distribution.

The effectiveness of detection hinges on the nonconformity measure, a heuristic function that scores how atypical a data point is. This function is trained solely on normal data.

• Distance-Based: Uses the distance to the k-nearest neighbors in the calibration set. A large distance implies high nonconformity.
• Density-Based: Employs the negative log-likelihood from a density estimator (e.g., Gaussian Mixture Model) fitted to normal data.
• Reconstruction-Based: Leverages the reconstruction error of an autoencoder or a Conformalized Autoencoder trained on normal samples. High error indicates an anomaly.
• One-Class Classification: Uses the decision function of a one-class SVM or similar model as the nonconformity score.

Unlike many heuristic anomaly detection methods, conformal anomaly detection provides a marginal guarantee on the false positive rate. If the calibration and test data are exchangeable, the probability of falsely flagging a normal point as anomalous is exactly bounded by the chosen significance level.

• No Asymptotics: The guarantee holds for any finite calibration set size, not just in the limit of infinite data.
• Model-Agnostic: The guarantee wraps around any underlying nonconformity measure, correcting its raw scores.
• Practical Impact: This allows engineers to set a precise, verifiable budget for false alarms in production monitoring systems.

A powerful modern instantiation combines deep learning with statistical rigor. A Conformalized Autoencoder uses the reconstruction error on a held-out calibration set of normal data to establish a valid threshold.

• Process: Train an autoencoder on normal data. Compute reconstruction errors on a separate calibration set. For a new point, compute its error and the corresponding conformal p-value.
• Advantage: Corrects for the autoencoder's potential overfitting or uneven reconstruction fidelity across the feature space.
• Output: A binary anomaly flag with a guaranteed upper bound on the false positive rate, making deep learning-based detection auditable.

Standard conformal anomaly detection controls the overall false positive rate. Mondrian Conformal Anomaly Detection extends this to provide conditional guarantees across pre-defined data slices or categories.

• Mechanism: Applies the conformal calibration procedure independently within each category (e.g., different machine types in a factory).
• Benefit: Ensures that the false positive rate is controlled not just on average, but also for each specific subgroup, preventing a model from performing poorly on a minority class of normal data.
• Use Case: Critical in industrial IoT where different equipment models have distinct normal operating signatures.

Conformal anomaly detection can be deployed in both static and streaming contexts. For non-stationary environments, Adaptive Conformal Inference techniques are essential.

• Offline (Split Conformal): A fixed calibration set defines a static anomaly threshold. Suitable for stable systems.
• Online (Adaptive): The quantile threshold is dynamically updated over time using a sliding window of recent observations. This maintains valid false positive rate control even under gradual distribution drift.
• Key Distinction: Online methods sacrifice the static threshold for robustness to changing normality, which is vital for monitoring evolving system metrics.

The foundational engine of conformal anomaly detection. A nonconformity measure is a heuristic function that scores how unusual a data point is relative to a calibration set of normal examples.

• In anomaly detection, this is often the reconstruction error of an autoencoder or the distance to the k-nearest neighbors.
• A higher score indicates a greater degree of strangeness.
• The choice of measure is the primary design decision; the conformal framework then wraps it to provide a statistically valid p-value.

The core statistical output used to flag anomalies. For a new test point, its conformal p-value is calculated as the proportion of calibration nonconformity scores that are greater than or equal to the test point's score.

• A small p-value (e.g., < 0.05) indicates the point is highly nonconforming relative to the normal data.
• Unlike classical hypothesis testing, these p-values are valid under the exchangeability assumption, providing finite-sample control over the false detection rate.
• They offer a rigorous, distribution-free alternative to heuristic thresholding.

A direct application of conformal anomaly detection to the out-of-distribution (OOD) problem. This framework tests whether a new input belongs to the training distribution.

• Uses conformal p-values to construct a statistical test with a guaranteed false positive rate.
• Unlike standard OOD methods that rely on arbitrary score cutoffs, conformal OOD detection provides a rigorous, user-specified Type I error control.
• It is model-agnostic and can wrap any pre-trained feature extractor or density estimator.

A specific technique that applies a conformal calibration step to the reconstruction error of an autoencoder for anomaly detection.

• The autoencoder is trained exclusively on normal (inlier) data.
• The reconstruction error serves as the nonconformity measure; anomalies are expected to have high reconstruction error.
• A calibration set of normal data is used to compute the empirical distribution of reconstruction errors, defining a statistically valid threshold that controls the false positive rate.

The fundamental assumption required for standard conformal anomaly detection to provide valid guarantees. Exchangeability means the joint distribution of the calibration and test data is invariant to any permutation.

• This is a weaker condition than the independent and identically distributed (IID) assumption.
• In anomaly detection, this assumes the calibration data of normal points and any future normal test points are exchangeable.
• Violations, such as temporal drift, require adaptive conformal inference or weighted conformal prediction to maintain validity.

An extension required when the exchangeability assumption is violated, such as in streaming data with concept drift. Adaptive conformal inference dynamically adjusts the threshold for flagging anomalies over time.

• Maintains long-run coverage or false positive rate guarantees without knowing the shift mechanism.
• Uses a rolling window of recent observations as the calibration set to handle temporal dependencies.
• Essential for deploying conformal anomaly detection in real-world, non-stationary production environments.