Glossary

Normalizing Flows

A generative modeling approach that transforms a simple base distribution into a complex one through a sequence of invertible mappings, enabling exact likelihood computation for OOD detection.

Get in touch Learn more

ML engineer managing model training cluster on laptop, GPU utilization visible, technical deep learning setup.

EXACT LIKELIHOOD GENERATIVE MODELING

What is Normalizing Flows?

A generative modeling approach that transforms a simple base distribution into a complex one through a sequence of invertible mappings, enabling exact likelihood computation for OOD detection.

Normalizing Flows are a class of generative models that learn a complex probability distribution by applying a sequence of invertible and differentiable transformations to a simple base distribution, such as a standard Gaussian. Unlike Variational Autoencoders or GANs, normalizing flows provide exact log-likelihood evaluation through the change-of-variables formula, making them uniquely suited for principled out-of-distribution detection where a calibrated probability density is required.

The architecture enforces that each transformation in the flow is bijective, meaning every point in the input space maps to a unique point in the latent space and vice versa. This constraint, typically satisfied by coupling layers or autoregressive transformations with easily computable Jacobian determinants, allows the model to directly compute p(x) without approximation. For OOD detection, this exact density serves as a rigorous anomaly score, flagging inputs that fall into low-likelihood regions of the learned manifold.

INVERTIBLE ARCHITECTURES

Core Properties of Normalizing Flows

Normalizing flows are a class of generative models that learn complex probability distributions by applying a sequence of invertible, differentiable transformations to a simple base distribution. Their defining characteristic is the ability to compute exact log-likelihoods, making them uniquely suited for principled out-of-distribution detection.

Exact Likelihood Computation

Unlike VAEs or GANs which rely on approximations or implicit modeling, normalizing flows provide tractable density estimation via the change-of-variables formula. For a bijective function f, the log-density is computed as log p_X(x) = log p_Z(f(x)) + log |det(J_f(x))|. This exactness allows for principled thresholding on probability values for OOD detection, where low-likelihood samples are flagged as anomalous.

Invertibility Constraint

Every transformation in a flow must be bijective—both forward mapping (inference) and inverse mapping (generation) must exist and be computationally efficient. This constraint ensures:

No information loss: Dimensionality is preserved throughout the network
Exact reconstruction: x = f^{-1}(f(x)) holds perfectly
Deterministic generation: Sampling from the base distribution and applying the inverse yields valid samples from the target distribution

Tractable Jacobian Determinant

The computational bottleneck in flows is calculating det(J_f(x)). Architectures are specifically designed with structured Jacobians to make this efficient:

Autoregressive flows: Triangular Jacobians where the determinant is the product of diagonal entries
Coupling layers: Block-diagonal structure where half the dimensions remain unchanged
Continuous flows: Use trace operations via Hutchinson's estimator instead of full determinants

Typicality vs. Density

A critical insight for OOD detection: high likelihood does not equal typicality. Normalizing flows can assign high density to inputs from the typical set—a concentrated region of high probability mass—while also assigning high density to out-of-distribution points far from this set. This motivates typicality tests that evaluate whether a sample resides in the typical set rather than relying solely on raw likelihood scores.

Multi-Scale Architecture

Modern flows employ squeeze and factor-out operations to handle high-dimensional data efficiently:

Squeeze: Reshapes spatial dimensions into channel dimensions, reducing resolution while preserving information
Factor-out: Splits dimensions at intermediate layers, modeling coarse features early and fine details later
Hierarchical modeling: Enables flows to scale to images like CelebA (256×256) by progressively building the distribution across multiple scales

Likelihood Regret for OOD

A robust OOD scoring mechanism that corrects for input complexity bias. Raw likelihoods can be misleading—complex images often receive lower likelihoods regardless of semantics. Likelihood regret computes LR(x) = log p(x) - log p_0(x), where p_0 is a background model trained on general data. This subtraction isolates whether the flow specifically recognizes the input as in-distribution, significantly improving detection of semantic anomalies.

NORMALIZING FLOWS EXPLAINED

Frequently Asked Questions

Clear, technically precise answers to the most common questions about normalizing flows, their mechanisms, and their role in out-of-distribution detection.

A normalizing flow is a generative model that transforms a simple base probability distribution, such as a standard Gaussian, into a complex target data distribution through a sequence of invertible and differentiable mappings. The core mechanism relies on the change of variables formula, which allows exact computation of the probability density of any data point by tracking how volume changes through each transformation. Starting with a random variable z0 drawn from a known base density p(z0), the flow applies a series of bijective functions f1, f2, ..., fK to produce x = fK ∘ ... ∘ f2 ∘ f1(z0). The log-likelihood of x is then computed as log p(x) = log p(z0) - Σ log |det(df_i/dz_{i-1})|, where the determinant of the Jacobian matrix accounts for the stretching or compression of space. This exact likelihood evaluation makes normalizing flows uniquely suited for out-of-distribution detection, as OOD inputs typically exhibit anomalously low likelihoods under the learned density model.

GENERATIVE ARCHITECTURE COMPARISON

Normalizing Flows vs. Other Generative Models

A technical comparison of Normalizing Flows against VAEs, GANs, and Diffusion Models for exact likelihood computation and OOD detection suitability.

Feature	Normalizing Flows	Variational Autoencoders	Generative Adversarial Networks	Diffusion Models
Exact Likelihood Computation
Invertible Mapping
Training Stability	High (direct MLE)	Moderate (ELBO)	Low (adversarial)	High (score matching)
Sample Quality	Moderate-High	Moderate	High	Very High
Inference Speed	Fast (single pass)	Fast (single pass)	Fast (single pass)	Slow (iterative denoising)
Latent Space Access
OOD Detection Suitability	Excellent (exact density)	Good (ELBO proxy)	Poor (no density)	Good (likelihood-based)
Mode Coverage	Full (by design)	Partial (KL gap)	Poor (mode collapse)	Full (by design)

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

NORMALIZING FLOWS

Applications in Out-of-Distribution Detection

Normalizing Flows provide a unique advantage in OOD detection by enabling exact likelihood computation. Unlike VAEs or GANs, they can directly calculate the probability density of an input, offering a mathematically rigorous threshold for rejecting anomalies.

Exact Log-Likelihood Scoring

The primary mechanism for OOD detection. Flows learn an invertible mapping $f: X \rightarrow Z$ from a complex data distribution to a simple base (e.g., Gaussian). Using the change of variables formula, the exact log-probability $\log p_X(x)$ is computed. In-distribution data receives high likelihood; OOD inputs receive low likelihood, providing a principled scalar score for rejection.

The Typicality Test Problem

A critical nuance. High likelihood does not always mean in-distribution. Flows can assign high density to nonsensical inputs (e.g., constant noise images) if they fall in a high-volume region of the base distribution. Typicality tests correct this by rejecting inputs that, despite high likelihood, fail to reside in the typical set of the model's learned manifold.

Likelihood Regret Mitigation

Flows can be biased by background statistics (e.g., image complexity). Likelihood Regret corrects for this by comparing the flow's likelihood to a generic background model's likelihood. An input is flagged as OOD if the flow does not provide a significantly better explanation than the background model, filtering out complex-but-irrelevant inputs.

Multi-Scale Flow Architectures

Architectures like RealNVP and Glow use multi-scale structures with squeeze and split operations. This allows the model to factor out local noise at different resolutions. For OOD detection, the likelihood contributions from coarse, semantic scales can be weighted more heavily, improving sensitivity to semantic anomalies rather than low-level texture differences.

Hybrid Discriminative-Generative Models

Combining flows with classifiers creates robust detectors. A class-conditional flow models $p(x|y)$, allowing for the computation of the posterior $p(y|x)$ via Bayes' rule. This joint framework provides both a class prediction and a confidence score from the marginal likelihood $p(x)$, enabling rejection of inputs that are far from all class-conditional manifolds.

Noise Contrastive Priors

Training flows with Outlier Exposure involves adding an auxiliary loss on a known OOD dataset. The model is trained to maximize likelihood on in-distribution data while minimizing it on outliers. This directly shapes the likelihood landscape, creating explicit low-probability valleys around the in-distribution manifold, significantly hardening the boundary against unknown attacks.

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.