Synthetic Data Vault (SDV)

SDV's Hierarchical Modeling Algorithm (HMA) extends single-table synthesis to entire relational databases. It learns and replicates primary key/foreign key relationships, ensuring referential integrity across parent and child tables.

• Generates synthetic versions of normalized financial databases
• Preserves complex joins between trade, quote, and corporate action tables
• Prevents referential integrity violations common in naive synthesis

The PAR Synthesizer models multi-sequence time-series data using a conditional probabilistic auto-regressive architecture. It captures complex temporal dependencies critical for financial data.

• Models volatility clustering and autocorrelation in asset returns
• Handles irregular sampling intervals common in tick data
• Conditions generation on known entity attributes like sector or market cap

SDV integrates a privacy evaluation framework that quantifies the risk of sensitive information leakage from synthetic datasets. It uses a combination of nearest-neighbor distance ratio metrics and adversarial disclosure risk scoring.

• Measures membership inference risk for individual records
• Provides numerical privacy scores alongside statistical fidelity scores
• Essential for generating compliant synthetic financial data under regulations

Users can define logical business rules that the synthetic data must obey. SDV's models learn to satisfy these constraints probabilistically, ensuring the generated data is not just statistically similar but logically coherent.

• Enforces domain rules like bid_price < ask_price
• Maintains column-level invariants and inter-column relationships
• Prevents generation of impossible financial scenarios

The SDMetrics library provides a comprehensive suite of statistical tests to evaluate synthetic data quality. It generates a detailed report comparing column shapes, pairwise correlations, and statistical moments between real and synthetic data.

• Quantifies the sim-to-real gap with numerical scores
• Detects mode collapse where the model ignores rare but critical tail events
• Visualizes distributional fidelity for stakeholder validation

The GaussianCopulaSynthesizer uses copula functions to model the complex dependence structure between columns independently of their marginal distributions. This is foundational for capturing fat-tail dependencies in financial returns.

• Separates marginal distribution learning from dependence structure learning
• Accurately models non-linear correlations between asset classes
• Handles mixed data types (continuous, categorical, datetime) in a single table

SDV models learn the joint distribution of Limit Order Book (LOB) levels, trade arrivals, and cancellations to generate synthetic market microstructure. This enables backtesting of execution algorithms against realistic, non-deterministic order flow without exposing proprietary exchange data. Key capabilities include:

• Replicating stylized facts like volatility clustering and fat tails
• Preserving Hawkes process self-excitation in trade arrivals
• Generating correlated multi-asset LOBs for portfolio execution testing

SDV-generated market environments serve as the training ground for Multi-Agent RL (MARL) systems. By conditioning generators on specific market regimes, quants create adversarial scenarios where trading agents compete against synthetic counterparts. This approach:

• Uses self-play to discover robust strategies without historical overfitting
• Generates market manipulation patterns like spoofing to test agent resilience
• Enables domain randomization across volatility and liquidity parameters

When real alternative datasets are sparse or expensive, SDV synthesizes correlated multi-table, multi-modal financial data. This includes generating synthetic corporate fundamentals, sentiment scores, and macroeconomic indicators that maintain realistic cross-sectional and temporal dependencies. Benefits include:

• Augmenting limited alpha factor research datasets
• Creating balanced training sets for rare event prediction
• Preserving copula structures between disparate data sources

SDV's Conditional GAN (CGAN) and copula models generate synthetic market paths conditioned on extreme events. Risk managers use these to evaluate portfolio vulnerability beyond historical crises. The approach models:

• Fat-tail distributions for asset returns with correct tail dependence
• Joint extreme moves across asset classes using vine copulas
• Regime-switching dynamics that transition between calm and stressed states

Financial institutions use SDV to share statistically faithful synthetic versions of proprietary trading data with quantitative research teams and third-party vendors. The Variational Autoencoder (VAE) and diffusion model backends ensure:

• Differential privacy guarantees against member inference attacks
• Preservation of stylized facts required for strategy development
• Compliance with data governance requirements while enabling collaboration

SDV generates synthetic execution traces to train market impact cost models without requiring millions of real order executions. By learning the conditional distribution of price moves given order size and market conditions, quants can:

• Build optimal execution algorithms that minimize slippage
• Test smart order routing logic across synthetic venue topologies
• Model the sim-to-real gap by comparing synthetic vs. live impact curves

SDV uses recursive conditional modeling to capture multi-table relational structures:

• Parent-child traversal: Learns dependencies between tables by modeling primary/foreign key relationships
• Referential integrity: Generated datasets maintain valid cross-table references automatically
• Sequential expansion: Models each table conditioned on its parent, preserving the original database schema
• Multi-table synthesis: Produces entire relational databases rather than isolated flat files, critical for realistic trading system backtesting

The GaussianCopulaSynthesizer models column distributions and their dependencies:

• Marginal distributions: Fits each column independently using parametric or non-parametric methods
• Covariance structure: Captures inter-column correlations through a Gaussian copula
• Mathematical foundation: Based on Sklar's theorem, which states any multivariate joint distribution can be expressed via copulas
• Financial application: Preserves the complex dependence structures between asset returns, volatility, and volume in synthetic market data

Conditional Tabular GAN addresses challenges specific to heterogeneous tabular data:

• Mode-specific normalization: Handles mixed data types (continuous, discrete, categorical) simultaneously
• Training-by-sampling: Conditionally generates rows to overcome class imbalance in minority categories
• Wasserstein loss variant: CTGAN can incorporate gradient penalty for improved training stability
• Financial use case: Generates realistic synthetic order book snapshots with proper distribution of limit order types and price levels

Time-series Generative Adversarial Network preserves temporal dynamics:

• Embedding network: Maps time series to a latent space capturing static and temporal features
• Recovery network: Ensures the latent representation faithfully reconstructs original sequences
• Joint training: Supervised loss on stepwise transitions combined with adversarial loss on sequence realism
• Market simulation: Generates synthetic price trajectories that maintain stylized facts like volatility clustering and fat-tail distributions

SDV provides SDMetrics for rigorous quality assessment:

• Statistical similarity: Kolmogorov-Smirnov tests compare column distributions between real and synthetic data
• Correlation preservation: Measures how well pairwise relationships are maintained
• Privacy metrics: Detects if synthetic records are too close to training samples, preventing memorization
• New row synthesis: Evaluates whether the model generates novel combinations rather than copying training rows
• Diagnostic reports: Automated HTML reports quantify data fidelity for compliance and model validation

Built-in mechanisms prevent sensitive information leakage:

• Differential privacy integration: Adds calibrated noise during model training to provide mathematical privacy guarantees
• Nearest neighbor distance ratio: Flags synthetic records that are suspiciously close to real training samples
• Column-level anonymization: Allows specifying which columns require enhanced privacy protection
• Regulatory compliance: Enables financial institutions to share realistic market datasets without exposing proprietary trading patterns or client information