Glossary

Novelty Detection

Novelty detection is the process of recognizing that a test input originates from a previously unseen pattern or class entirely absent from the training distribution.

Get in touch Learn more

Data scientist building training data pipeline on laptop, data preprocessing visible, technical workspace.

OUT-OF-DISTRIBUTION DETECTION

What is Novelty Detection?

Novelty detection is the machine learning task of identifying test inputs that originate from a previously unseen pattern or class entirely absent from the training distribution, enabling models to recognize when they encounter unknown concepts.

Novelty detection is the process of recognizing that a test input belongs to a new semantic class not represented in the training data. Unlike out-of-distribution detection, which flags any statistically divergent input, novelty detection specifically identifies emerging patterns—such as a new malware family or a previously unseen product defect—that should be isolated for further analysis rather than forced into an existing category.

This capability relies on modeling the support of the training distribution through techniques like Deep SVDD or Local Outlier Factor (LOF). By learning a compact boundary around known data in feature space, the system can reject inputs that fall outside this manifold, preventing the silent misclassification of novel phenomena and forming a critical component of open set recognition systems.

METHODOLOGICAL FOUNDATIONS

Core Novelty Detection Techniques

A taxonomy of the primary algorithmic approaches used to identify inputs that belong to previously unseen classes or patterns, distinct from anomaly detection which flags rare in-distribution events.

One-Class Classification

Trains a model exclusively on the target 'normal' class to define a tight decision boundary. Deep SVDD maps data into a minimal hypersphere, flagging points far from the center as novel. One-Class SVM finds a hyperplane that separates the origin from the training data in a high-dimensional kernel space. These methods are ideal when only positive samples are available and the space of potential novelties is unbounded.

Distance-Based Rejection

Uses the distance of a test sample's feature representation to the training manifold as a novelty score. Mahalanobis Distance computes the distance to class-conditional Gaussian distributions, capturing covariance structure. KNN Distance uses the distance to the k-th nearest neighbor in the training feature space. Inputs far from the learned manifold are rejected as novel.

Uncertainty Quantification

Leverages model uncertainty as a proxy for novelty. Epistemic uncertainty—the model's lack of knowledge—is high for unseen inputs. Techniques include:

Monte Carlo Dropout: Multiple forward passes with dropout at inference time
Deep Ensembles: Variance across independently trained models High predictive disagreement signals a novel input outside the training distribution.

Generative Model Likelihood

Uses the probability density assigned by a generative model as a normality score. Normalizing Flows provide exact likelihood computation through invertible transformations. Diffusion Models detect novelty via high reconstruction error on noised inputs. However, likelihood-based methods can fail on complex datasets—a phenomenon known as the typicality paradox, where OOD inputs receive higher likelihoods than in-distribution samples.

Contrastive Representation Learning

Trains encoders to pull augmented views of the same sample together while pushing all others apart in embedding space. This creates a feature manifold where novel classes naturally separate from known ones. Methods like SimCLR and SupCon improve OOD detection by learning representations that are sensitive to semantic shifts, not just pixel-level differences.

Outlier Exposure Training

A training strategy that leverages an auxiliary dataset of diverse outliers to teach the model heuristics for detecting unknown inputs. The model is trained to produce uniform softmax distributions or high energy scores on outlier samples. This significantly improves generalization to unseen novelty distributions and is one of the most empirically effective approaches when a representative outlier set is available.

TAXONOMIC COMPARISON

Novelty Detection vs. Related Concepts

Distinguishing novelty detection from adjacent paradigms in out-of-distribution analysis based on training assumptions, output semantics, and operational objectives.

Feature	Novelty Detection	Anomaly Detection	Open Set Recognition
Training Data Composition	Only normal/in-distribution samples; no anomalies available	Mostly normal data; may contain rare contaminated outliers	Labeled known classes only; unknown classes entirely absent
Primary Objective	Identify inputs from previously unseen classes or patterns	Identify rare events or statistical deviations from the norm	Classify known classes accurately while rejecting unknown classes
Output Semantics	Binary: novel vs. normal; no class discrimination required	Continuous anomaly score or binary outlier flag	Multi-class prediction plus explicit 'unknown' rejection class
Known Class Discrimination
Handles Contaminated Training Data
Typical Supervision Paradigm	Semi-supervised (one-class training)	Unsupervised or semi-supervised	Supervised for known classes; open-set at inference
Core Statistical Assumption	Novel patterns are absent from training distribution entirely	Anomalies are rare and lie in low-density regions	Known and unknown classes are semantically disjoint
Representative Algorithms	Deep SVDD, One-Class SVM, Autoencoder Reconstruction Error	Isolation Forest, LOF, KNN Distance, Robust PCA	OpenMax, Extreme Value Theory calibration, PROSER

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

NOVELTY DETECTION INSIGHTS

Frequently Asked Questions

Explore the core concepts, methodologies, and distinctions that define how machine learning systems identify previously unseen patterns and classes absent from their training distribution.

Novelty detection is the process of recognizing that a test input originates from a previously unseen pattern or class that was entirely absent from the training distribution. While often used interchangeably, novelty detection and anomaly detection have a critical semantic distinction in machine learning. In novelty detection, the training set is assumed to be clean and consists solely of 'normal' in-distribution data; the goal is to detect new, previously unobserved patterns that emerge later. In anomaly detection, the training set may already contain outliers or anomalies, and the task is to identify these rare, deviant instances within a contaminated dataset. Novelty detection is fundamentally an open set recognition problem where the model must generalize its concept of 'known' to reject the 'unknown' without having been explicitly trained on outlier examples.

NOVELTY DETECTION ECOSYSTEM

Related Terms

Novelty detection intersects with a broad family of techniques for identifying unfamiliar inputs. These related concepts span statistical methods, uncertainty quantification, and open-set classification frameworks.

Out-of-Distribution (OOD) Detection

The broader task of identifying inputs that are semantically or statistically different from the training distribution. While novelty detection focuses on recognizing previously unseen classes, OOD detection encompasses any input that should not be processed by the model.

Semantic OOD: Inputs from entirely different domains (e.g., a cat photo fed to a medical imaging model)
Covariate Shift: Inputs from the same domain but with different statistical properties
Key distinction: Novelty detection is a subset of OOD detection specifically concerned with new classes

EXPLORE

Open Set Recognition

A classification framework requiring models to accurately classify known classes while simultaneously rejecting samples from unknown classes not seen during training. Unlike closed-set classification, open set recognition explicitly models the possibility of novelty.

OpenMax replaces softmax with Weibull-calibrated activation vectors
Compact Abating Probability models the risk of unknown classes
Critical for real-world deployment where encountering novel classes is inevitable

EXPLORE

Epistemic vs. Aleatoric Uncertainty

Two fundamental types of uncertainty that inform novelty detection strategies:

Epistemic Uncertainty: Model uncertainty from lack of knowledge — reducible with more training data. High epistemic uncertainty signals novel inputs outside the training distribution.
Aleatoric Uncertainty: Irreducible data uncertainty from noise, class overlap, or inherent stochasticity. Cannot be reduced by collecting more samples.
Monte Carlo Dropout and Deep Ensembles estimate epistemic uncertainty by measuring prediction variance across multiple forward passes or independently trained models.

One-Class Classification Methods

Techniques that model only the normal or known class distribution, flagging anything outside as novel or anomalous:

Deep SVDD: Trains a neural network to map normal data into a minimal hypersphere, flagging points far from the center
Isolation Forest: Exploits the fact that anomalies are easier to isolate through random feature splits
Local Outlier Factor (LOF): Measures local density deviation relative to neighbors
These methods are particularly useful when anomalous examples are unavailable during training

Post-Hoc Detection Scores

Lightweight methods applied to already-trained models without retraining, using internal activations or outputs as novelty signals:

Maximum Softmax Probability (MSP): Uses the highest softmax score as a confidence measure — simple but often overconfident on OOD inputs
Energy-Based Scores: Uses Helmholtz free energy from logits; lower energy indicates in-distribution
ReAct: Rectifies activations by clipping extreme values to reduce OOD overconfidence
GradNorm: Exploits higher gradient magnitudes of KL divergence for in-distribution data
ViM: Combines feature norms with logits using principal component residual projection

Training-Time Regularization Strategies

Techniques applied during model training to improve novelty detection capability:

Outlier Exposure: Leverages an auxiliary dataset of outliers to teach heuristics for detecting unknown inputs — significantly improves generalization to unseen OOD distributions
Contrastive Training: Pulls augmented views of the same sample together while pushing others apart, improving feature separability
Virtual Adversarial Training (VAT): Smooths output distributions against local perturbations, improving softmax score uniformity for OOD inputs
Hyperspherical Embedding: Constrains features to a unit sphere, reducing feature collapse and improving class separation

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.