Nearest Neighbor Adversarial Accuracy (NNAA)

NNAA operates on a simple geometric principle. For any synthetic record, it calculates the distance to its nearest neighbor in the synthetic set (d_S) and its nearest neighbor in the real training set (d_R). The ratio d_S / d_R determines if the record is 'privacy-safe' or 'overfitted'.

• Ratio > 1: The synthetic point is closer to other synthetic points (safe).
• Ratio ≈ 0: The synthetic point is a near-copy of a real record (high privacy risk).
• Threshold: Typically, a ratio < 0.3 indicates a potential training data copy.

The final NNAA score is the accuracy of a hypothetical privacy attacker. The attacker guesses a record is 'real' if its nearest neighbor in the combined (real + synthetic) space is also real.

• NNAA = 1.0: Perfect attacker accuracy. The synthetic data is trivially distinguishable and likely contains memorized copies.
• NNAA ≈ 0.5: Ideal privacy. The attacker cannot distinguish real from synthetic data any better than a random coin flip.
• NNAA < 0.5: The synthetic data is 'too distinct' from the real distribution, indicating poor utility.

NNAA is a direct empirical measure of vulnerability to Membership Inference Attacks (MIAs). While differential privacy provides a theoretical epsilon guarantee, NNAA provides a post-hoc empirical test.

• Black-Box Assessment: It requires no access to the generator's internal weights or loss function.
• Identifies Overfitting: A high NNAA score directly correlates with the generator memorizing rare training examples, such as outliers in clinical datasets.
• Complementary Metric: Use NNAA alongside Differential Privacy (DP) bounds to provide both theoretical and empirical privacy assurance.

NNAA is rarely used in isolation; it is plotted against utility metrics like the Train on Synthetic, Test on Real (TSTR) score to find the Pareto frontier.

• High Utility, High NNAA: The model is copying real data. This is a failed privacy outcome.
• Low Utility, Low NNAA: The model is generating noise. This is a failed utility outcome.
• The Sweet Spot: A configuration where NNAA is close to 0.5 and downstream classification accuracy remains high. This validates the generator's ability to learn the manifold without memorizing points.

In high-dimensional spaces (e.g., genomics or imaging), distance metrics suffer from the curse of dimensionality, where all points appear equidistant.

• Feature Reduction: NNAA is often computed on a lower-dimensional latent representation (e.g., extracted via a pre-trained autoencoder) rather than raw pixels or base pairs.
• Semantic Similarity: For images, using Euclidean distance on pixels is meaningless. NNAA should be applied to embeddings from a Frechet Inception Distance (FID)-style feature extractor to measure semantic, not pixel-level, copying.

Integrating NNAA into a synthetic data pipeline involves a hold-out validation step:

1. Split Data: Divide the real dataset into a Training Set (used for the generator) and a Hold-out Set (unseen by the generator).
2. Generate: Create a synthetic dataset of equal size to the hold-out set.
3. Combine: Merge the synthetic data with the real hold-out data.
4. Classify: For each record in the combined set, check if the nearest neighbor is from the real or synthetic pool.
5. Calculate: NNAA = (Correctly identified real records) / (Total real records).

A composite metric evaluating synthetic data across three dimensions: statistical fidelity, downstream utility, and privacy protection. NNAA serves as the primary privacy component in many quality scoring frameworks.

• Fidelity: Distributional similarity to real data
• Utility: Performance preservation in ML tasks
• Privacy: NNAA and related distance-based metrics
• Provides a holistic view for regulatory submissions

A technique for detecting distribution shift by training a classifier to distinguish between two datasets. NNAA operationalizes this concept specifically for privacy evaluation, using nearest neighbor classifiers as the discriminator.

• Reveals whether synthetic data is distinguishable from real data
• High adversarial accuracy signals poor privacy or fidelity
• Used iteratively during GAN training to monitor convergence

The degree to which synthetic medical data adheres to physiological constraints and medical ontologies like SNOMED CT. While NNAA measures privacy, clinical plausibility ensures generated records remain medically coherent.

• Validates realistic disease progression pathways
• Ensures lab values fall within biologically possible ranges
• Balances privacy metrics with clinical utility requirements
• Essential for FDA and EMA regulatory acceptance