Bayesian Network

A Bayesian network represents variables as nodes and conditional dependencies as directed edges, forming a directed acyclic graph (DAG). This structure encodes the Markov condition: each node is conditionally independent of its non-descendants given its parents. The acyclic constraint ensures no feedback loops, enabling efficient factorization of the joint probability distribution into a product of local conditional probability tables (CPTs).

• Nodes: Random variables (e.g., symptoms, diseases, biomarkers)
• Edges: Direct probabilistic influence
• CPTs: Quantify P(child | parents) for discrete variables

The primary computational advantage of Bayesian networks is their ability to exploit conditional independence to dramatically reduce the number of parameters needed to specify a joint distribution. Without independence assumptions, representing n binary variables requires 2^n - 1 parameters. By encoding only direct dependencies, a Bayesian network reduces this to a sum over local CPTs. D-separation is the graphical criterion used to read off all conditional independencies implied by the graph structure.

• D-separation: Determines if nodes are conditionally independent given evidence
• Parameter reduction: From exponential to linear in graph connectivity
• Sparse graphs: Indicate strong independence assumptions

Bayesian networks support two types of probabilistic inference: exact inference using variable elimination or junction tree algorithms, and approximate inference via Markov Chain Monte Carlo (MCMC) or belief propagation. Exact inference is NP-hard in general but tractable for networks with low treewidth. For synthetic data generation, inference computes the posterior distribution over unobserved variables, enabling ancestral sampling from the learned joint distribution.

• Variable elimination: Sums out non-query variables sequentially
• Junction tree: Compiles graph into a tree of cliques for efficient marginalization
• Likelihood weighting: Importance sampling that weights samples by evidence probability

Learning a Bayesian network from data involves two tasks: parameter learning (estimating CPTs given a known graph structure) and structure learning (discovering the graph itself). Parameter learning uses maximum likelihood estimation or Bayesian estimation with Dirichlet priors. Structure learning employs score-based methods (BIC, BDeu scores) with heuristic search, or constraint-based methods using conditional independence tests. Hybrid algorithms like MMHC combine both approaches for scalability.

• Complete data: Closed-form MLE for CPTs
• Incomplete data: Expectation-Maximization (EM) algorithm
• Structure search: Hill-climbing, tabu search over DAG space

When edges represent direct causal relationships, a Bayesian network becomes a causal Bayesian network supporting do-calculus and interventional queries. The do-operator simulates setting a variable to a specific value by removing incoming edges, distinguishing P(Y | X=x) from P(Y | do(X=x)). This enables counterfactual reasoning—predicting what would have happened under a different intervention—critical for generating synthetic data that preserves causal structure rather than mere correlations.

• Do-calculus: Three rules for transforming interventional distributions
• Pearl's back-door criterion: Identifies sufficient adjustment sets
• Counterfactuals: Require functional causal models with exogenous noise variables

To generate synthetic data, a learned Bayesian network performs ancestral sampling: nodes are sampled in topological order, with each node drawn from its conditional distribution given already-sampled parents. This produces i.i.d. samples from the joint distribution P(X₁,...,Xₙ). For conditional generation, evidence nodes are clamped to desired values before sampling remaining variables. The generated data preserves marginal distributions, conditional dependencies, and—if causally structured—interventional distributions.

• Topological ordering: Ensures parents are sampled before children
• Forward sampling: Draws from root nodes first, propagating downward
• Rejection sampling: Discards samples inconsistent with evidence constraints