Normalizing Flow

The core mechanism of a normalizing flow is a sequence of bijective transformations (f = f_K ∘ ... ∘ f_1). Each function f_i must be invertible and differentiable, with an easily computable Jacobian determinant. This ensures that the mapping from the base distribution p(z) to the target distribution p(x) is a one-to-one correspondence, allowing exact density evaluation via the change of variables formula:

• log p(x) = log p(z) - Σ log |det(df_i/dz_{i-1})|
• This property distinguishes flows from VAEs and GANs, which only provide implicit or bounded likelihood estimates.

A widely used architectural building block that ensures efficient invertibility and a tractable Jacobian. A coupling layer splits the input vector into two parts (x_a, x_b).

• Forward pass: x_a is passed through unchanged, while x_b is transformed by a scale-and-shift network conditioned on x_a.
• Inverse pass: The process is reversed analytically without needing to invert the neural network.
• Jacobian: The transformation matrix is triangular, making the log-determinant simply the sum of the log-scale coefficients.
• Implementations like RealNVP and Glow use this to stack deep, expressive flows while maintaining O(D) computational complexity.

A class of normalizing flows where the transformation of each dimension depends only on previous dimensions, enforcing a triangular Jacobian by design.

• Masked Autoregressive Flow (MAF): Uses masked dense networks to ensure the autoregressive property, making density estimation fast but sampling inherently sequential and slow.
• Inverse Autoregressive Flow (IAF): Reverses the role, enabling fast parallel sampling but slow density evaluation.
• Parallel Wavenet: Distills an IAF 'student' from a MAF 'teacher' to achieve both fast training and low-latency sampling, a critical technique for real-time audio generation.

Instead of a discrete sequence of transformations, a CNF defines the dynamics via an ordinary differential equation parameterized by a neural network: dz/dt = f(z(t), t; θ).

• Instantaneous Change of Variables: The log-density evolves according to the trace of the Jacobian: d log p(z(t))/dt = -Tr(df/dz).
• FFJORD (Free-form Jacobian of Reversible Dynamics): Uses Hutchinson's trace estimator to compute the trace efficiently, removing architectural constraints like coupling layers.
• This formulation allows for continuous-depth models and adaptive computation, trading off exact architectural control for maximum expressiveness.

A technique for building invertible residual networks by enforcing the Lipschitz constant of the residual block to be less than 1.

• Banach Fixed-Point Theorem: If the residual function g(x) is a contraction mapping (Lip(g) < 1), the transformation f(x) = x + g(x) is guaranteed to be invertible.
• The inverse can be found via a fixed-point iteration, avoiding the need for an analytical inverse.
• i-ResNet: Uses spectral normalization and a specialized log-determinant estimator based on a power series expansion of the matrix logarithm, enabling unrestricted ResNet-style architectures to be used as normalizing flows.

The defining advantage of normalizing flows over other generative models. Because the mapping is bijective and the Jacobian is tractable, the model provides an exact, closed-form computation of log p(x).

• This enables direct optimization of the log-likelihood on training data, a stable and well-understood objective.
• Out-of-Distribution Detection: Exact densities allow for principled anomaly detection by thresholding likelihood values.
• Lossless Compression: Flows can be used with bits-back coding to achieve near-optimal compression rates, as demonstrated by models like IDF (Integer Discrete Flows).

Unlike GANs or VAEs, normalizing flows provide tractable exact log-likelihoods by design through the change-of-variables formula. This is critical for modeling rare disease phenotypes where accurate probability estimation directly impacts diagnostic screening thresholds.

• Directly computes p(x) for any patient record without approximation
• Enables statistically rigorous anomaly detection for outlier clinical presentations
• Supports maximum likelihood training with guaranteed convergence properties
• Facilitates principled model comparison using held-out log-likelihood scores

The defining property of normalizing flows is the bijective transformation between a simple latent distribution and complex clinical data space. This invertibility enables both efficient sampling and exact inference in a single architecture.

• Forward direction: encode real patient data into structured latent representations for clustering
• Reverse direction: sample from the latent space to generate novel synthetic patients
• Guarantees one-to-one correspondence between latent codes and generated records
• Enables latent space interpolation between patient phenotypes for data augmentation

Normalizing flows enable precise measurement of how likely a generated record is to have originated from the training distribution, providing a mathematical basis for privacy risk quantification.

• Compute exact probability of synthetic records matching real training samples
• Apply differential privacy by clipping and noising the gradient of the log-likelihood
• Detect and filter generated records that fall too close to training data in density space
• Combine with membership inference attack resistance metrics for formal privacy auditing

Modern flow architectures like RealNVP and Masked Autoregressive Flows incorporate specialized coupling layers that handle the heterogeneous data types common in electronic health records.

• Continuous variables: lab values, vital signs modeled with smooth transformations
• Discrete variables: diagnosis codes, medication orders using dequantization techniques
• Ordinal variables: pain scores, disease staging via cumulative distribution functions
• Categorical variables: demographic fields through simplex-based flows

Autoregressive normalizing flows extend the framework to sequential data, enabling generation of realistic disease progression timelines while maintaining temporal consistency.

• Model conditional distributions p(xt | x<t) for each time step in a patient history
• Capture irregular sampling patterns common in real-world clinical data
• Generate counterfactual trajectories by manipulating specific latent dimensions
• Preserve clinical plausibility across longitudinal lab trends and medication sequences

Normalizing flows can be conditioned on structured medical knowledge, enabling generation of synthetic patients that respect ontological constraints from resources like SNOMED CT and ICD-11.

• Condition generation on disease hierarchies to ensure diagnostic coherence
• Enforce physiological constraints through constrained latent space transformations
• Generate patients with specific comorbidity profiles for targeted clinical trial simulation
• Combine with graph neural networks to incorporate relational clinical knowledge

The mathematical foundation of normalizing flows, expressing how probability density transforms under invertible mappings. For a bijective function f mapping z to x:

• p(x) = p(z) |det(J_f⁻¹(x))| where J is the Jacobian matrix
• The determinant of the Jacobian captures local volume change
• Flows require tractable Jacobian determinants for efficient training
• Architectures like RealNVP and Glow design transformations with triangular Jacobians for O(n) determinant computation

A composite metric evaluating synthetic data across three critical dimensions:

• Statistical fidelity: How well marginal and joint distributions match the original data
• Utility: Performance of downstream models trained on synthetic data when tested on real data
• Privacy: Resistance to membership inference and attribute disclosure attacks Normalizing flows excel in fidelity metrics due to exact likelihood optimization, making them strong candidates for privacy-preserving medical data generation.

An evaluation paradigm measuring synthetic data utility by training models exclusively on generated data and testing on held-out real data. Key considerations:

• A TSTR score close to Train-Real-Test-Real (TRTR) performance indicates high utility
• Normalizing flows trained via maximum likelihood often achieve strong TSTR results
• Particularly relevant for clinical prediction models where real training data is inaccessible due to privacy constraints
• Complements privacy metrics like NNAA for holistic synthetic data assessment

A generative model incorporating causal structure and do-calculus to generate counterfactual data points. When combined with normalizing flows:

• Flows model the joint distribution of cause-effect pairs
• Enable simulation of interventions by manipulating latent variables
• Support counterfactual patient generation for treatment effect estimation
• Remove confounding biases through explicit causal graph constraints
• Critical for generating synthetic clinical trial data with realistic treatment responses