t-SNE Visualization

t-SNE begins by modeling pairwise similarities in the high-dimensional data space. For each data point i, it constructs a conditional probability distribution over all other points j, where the probability that j is a neighbor of i is proportional to a Gaussian kernel centered at i. This probability p_{j|i} is high for nearby points and vanishingly small for distant points. The algorithm symmetrizes these probabilities to create a joint probability distribution P over all pairs, representing the local structure of the original data.

• Key Parameter: The perplexity controls the effective number of local neighbors considered for each point, balancing attention to local versus global structure.

In the low-dimensional (2D or 3D) embedding space, t-SNE defines a second probability distribution Q over pairs of mapped points. Crucially, it uses a heavy-tailed Student-t distribution with one degree of freedom (a Cauchy kernel) to model these similarities. This choice is fundamental: the fat tails of the t-distribution allow mapped points to be moderately far apart in the low-dimensional space without incurring a large cost, which prevents "crowding"—the tendency of points to collapse into the center of the map.

• Result: Distant points in high dimensions can be placed far apart, while local neighborhoods are preserved, creating clear, separated clusters in the visualization.

The goal of t-SNE is to learn a low-dimensional embedding where the distribution Q closely matches the high-dimensional distribution P. It achieves this by minimizing the Kullback-Leibler (KL) Divergence between the two distributions using gradient descent. The cost function is:

C = KL(P || Q) = Σ_i Σ_j p_{ij} log( p_{ij} / q_{ij} )

The gradient of this cost with respect to the positions of the low-dimensional points has an intuitive physical interpretation: it creates attractive forces between points with high p_{ij} (similar points) and repulsive forces between points with low p_{ij}. The optimization iteratively adjusts point positions to settle into a stable, low-dimensional representation.

When validating synthetic data, t-SNE visualizations are used to compare the joint distributions of real and generated samples. A well-generated synthetic dataset should be visually indistinguishable from the real data in the t-SNE plot.

• Ideal Outcome: Real and synthetic points are interleaved within the same cluster structures, indicating matched local and global distributions.
• Warning Signs: Separate clusters for real vs. synthetic data indicate a distribution shift. Gaps or holes within a cluster suggest the synthetic data lacks diversity or has mode collapse. Synthetic points forming tight, isolated blobs may indicate over-regularization or lack of natural variance.

t-SNE is a visualization tool, not a quantitative metric. Key limitations must be understood:

• Non-Global Structure: Distances between well-separated clusters in the plot are not meaningful. A large gap between two clusters does not necessarily reflect their true high-dimensional separation.
• Stochasticity: Different runs with different random seeds can produce different layouts, though cluster membership should be consistent.
• Hyperparameter Sensitivity: The perplexity parameter significantly affects the result. Low perplexity focuses on very local structure; high perplexity considers more global neighbors. It must be tuned for the dataset.
• Scale: The algorithm does not preserve density. A dense cluster in high dimensions may appear larger or smaller in the plot.
• Computational Cost: The pairwise computation is O(N²), making it slow for very large datasets (though approximations like Barnes-Hut t-SNE exist).

t-SNE should always be used alongside quantitative validation metrics to form a complete assessment of synthetic data fidelity.

• Distribution-Level Metrics: Use Fréchet Inception Distance (FID) or Kernel Inception Distance (KID) for images, or Maximum Mean Discrepancy (MMD) for general data to measure distribution similarity.
• Utility Metrics: Employ Train-on-Synthetic Test-on-Real (TSTR) to measure the downstream usefulness of the data.
• Classifier-Based Tests: Adversarial validation or a domain classifier can provide a scalar score for distinguishability.

t-SNE provides the "why" behind a poor quantitative score, revealing the structural nature of any discrepancy.

The primary visual signal in a t-SNE plot is the formation of clusters. For synthetic data validation, inspect whether real and synthetic data points for the same class form a single, cohesive cluster or remain separate.

• Good sign: Real and synthetic points for the same class are intermixed within a single cluster.
• Warning sign: Real and synthetic points for the same class form distinct, separate sub-clusters, indicating a distribution shift where the synthetic data does not accurately mimic the real data's local structure.
• Critical failure: Synthetic points form a large, amorphous blob distinct from all real data clusters, suggesting mode collapse or extremely low fidelity.

t-SNE preserves local structure but distorts global distances. This is crucial for correct interpretation.

• Distances between clusters are meaningless. A large gap between two clusters does not necessarily mean the data classes are dissimilar in the original high-dimensional space.
• Focus on local neighborhoods. The algorithm's goal is to place similar points close together. Therefore, the proximity of a synthetic point to its real counterparts within a local neighborhood is a meaningful indicator of fidelity.
• Avoid quantitative conclusions from gap sizes. Do not use the measured distance between cluster centroids in the t-SNE plot as a quantitative metric. It is not a reliable measure of distribution similarity like Wasserstein Distance or Maximum Mean Discrepancy (MMD).

Perplexity is t-SNE's most important hyperparameter. It can be thought of as a guess for the number of close neighbors each point has. Changing it dramatically alters the plot.

• Low Perplexity (e.g., 5-15): Emphasizes local structure. Many small, tight clusters will appear. Useful for checking fine-grained fidelity within a class.
• High Perplexity (e.g., 30-50): Emphasizes global structure. Fewer, larger clusters form. Better for seeing broader class separation.
• Validation Protocol: Always generate and compare multiple t-SNE plots across a range of perplexity values (e.g., 5, 30, 50). A robust synthetic dataset should show good mixing of real and synthetic points across all perplexity settings. If mixing only occurs at one specific perplexity, the fidelity may be fragile.

t-SNE plots can be misleading. Recognize these common artifacts to avoid false conclusions.

• Crowding Problem: t-SNE struggles to place points uniformly in 2D, often creating artificial-looking central gaps or "doughnut" shapes within a cluster. Do not interpret these gaps as meaningful separations.
• Stochastic Nature: Different runs with the same parameters and data will produce visually different layouts. Always use a fixed random seed for reproducible validation comparisons.
• Sample Size Sensitivity: t-SNE can be unstable with very small sample sizes. Use a sufficient, matched number of real and synthetic samples (e.g., 1,000-5,000 per set) for a stable visualization.
• Not a Substitute for Metrics: t-SNE is an exploratory tool, not a definitive test. It must be complemented with quantitative metrics like Fréchet Inception Distance (FID), Precision and Recall for Distributions (P&R), and Downstream Task Performance.

Integrate t-SNE visualization into a systematic synthetic data validation pipeline.

1. Feature Extraction: Compute embeddings for both real and synthetic data using a relevant model (e.g., a pre-trained vision backbone for images, the last layer of a classifier for tabular data).
2. Plot Generation: Create labeled t-SNE plots (color=class, shape=real/synthetic) across multiple perplexities.
3. Qualitative Assessment: Visually score for cluster mixing, separation, and the absence of artifacts.
4. Quantitative Corroboration: Use the same embeddings to calculate adversarial validation accuracy (a domain classifier) or Maximum Mean Discrepancy (MMD). The qualitative t-SNE findings should align with these quantitative scores.
5. Iterative Feedback: Use t-SNE insights (e.g., "class X synthetic data is forming its own cluster") to debug and refine the synthetic data generation process.

t-SNE should never be used in isolation. Combine it with these other visualization and validation methods for a complete assessment.

• UMAP: Often provides better preservation of global structure than t-SNE and is faster. Use UMAP plots as a consistency check.
• PCA: A simple linear projection. If real and synthetic data separate in PCA but mix in t-SNE, it suggests the distributions differ in variance/covariance (captured by PCA) but share complex local manifolds (captured by t-SNE).
• Domain Classifier Confidence: Plot the histogram of confidence scores from a domain classifier (real vs. synthetic). A classifier with near-50% confidence for all samples indicates good mixing, aligning with a well-mixed t-SNE plot.
• Per-Class Feature Distribution Plots: For key features, plot the distribution (histogram/KDE) for real vs. synthetic data per class to identify specific attribute-level discrepancies hinted at by t-SNE.

Maximum Mean Discrepancy (MMD) is a kernel-based statistical test used to determine if two samples (e.g., real and synthetic data) are drawn from the same distribution. It computes the distance between the means of the two samples after mapping them into a high-dimensional Reproducing Kernel Hilbert Space (RKHS).

• Role in Validation: Provides a rigorous, quantitative measure of distributional similarity that complements t-SNE's visual intuition.
• Key Difference: While t-SNE offers a 2D/3D visualization of perceived similarity, MMD provides a formal statistical test for distributional equivalence.

Fréchet Inception Distance (FID) is a cornerstone metric for evaluating synthetic image quality. It calculates the Wasserstein-2 distance between the multivariate Gaussian distributions fitted to the feature activations of real and generated images, as extracted by a pre-trained Inception-v3 network.

• Visual Correlation: A low FID score indicates the feature distributions are close, which should correspond to tight intermingling of real and synthetic points in a t-SNE plot of those same features.
• Standard Benchmark: The de facto standard for benchmarking generative models like GANs and diffusion models.

Precision and Recall for Distributions (P&R) is a two-dimensional metric that separately measures the quality and diversity/coverage of a synthetic dataset.

• Precision: The fraction of synthetic samples that fall within the support of the real data distribution (are realistic).
• Recall: The fraction of real data modes that are covered by the synthetic data.
• Diagnostic Power: A t-SNE visualization can illustrate P&R failures: low precision appears as synthetic outliers; low recall appears as empty regions in the real data cluster.

A Domain Classifier is a discriminative model (e.g., a simple neural network) trained to distinguish between samples from the real and synthetic datasets. Its performance is a direct proxy for data fidelity.

• Validation Metric: If the classifier achieves near-random (e.g., 50%) accuracy, it indicates the synthetic data is statistically indistinguishable from the real data.
• Link to t-SNE: A successful domain classifier would find a separable boundary between the two sample types. In a good t-SNE plot, this boundary should not exist, with points from both domains intermixed.

Train-on-Synthetic Test-on-Real (TSTR) is the ultimate utility evaluation protocol. It measures the practical value of synthetic data by training a downstream ML model (e.g., a classifier) entirely on the synthetic set and evaluating its performance on a held-out set of real data.

• Primary Goal: High TSTR performance proves the synthetic data preserves the feature-label relationships necessary for model training.
• Visual Precursor: t-SNE can provide an early, qualitative signal. If real and synthetic data are well-aligned in the latent space used for t-SNE, it suggests the feature representations are similar, which is a prerequisite for good TSTR performance.

Statistical Parity (Demographic Parity) is a fairness metric requiring a model's outcomes to be independent of protected attributes (e.g., race, gender). Bias Amplification occurs when a model exacerbates societal biases present in its training data.

• t-SNE's Role: Visualizing synthetic data by protected attribute can reveal clustering or separation that indicates the generation process may be replicating or distorting underlying biases.
• Proactive Audit: t-SNE serves as an initial audit tool before quantitative fairness metrics are calculated, helping to identify potential equity issues in the synthetic dataset.