Data Valuation

A game-theoretic approach that fairly distributes model performance credit among training data points. Data Shapley computes the average marginal contribution of a data point across all possible subsets of the training set.

• Equitable valuation: Satisfies axioms of fairness including symmetry, null player, and additivity
• Computational challenge: Exact calculation requires 2^N model retrainings, necessitating Monte Carlo approximation
• Truncation methods: Gradient Shapley and TMC-Shapley reduce complexity by early stopping permutations
• Application: Identifies high-value records for targeted synthetic augmentation and low-value records for pruning

A first-order approximation of data value measured by retraining the model without a specific data point and observing the change in performance. LOO influence is computationally tractable but fails to capture complex interactions between data points.

• Influence functions: Provide an analytical approximation without full retraining by estimating parameter changes via the Hessian
• Cook's distance: A classical statistical analog measuring the influence of individual observations on regression coefficients
• Limitation: Cannot detect redundant or highly correlated data points that individually have low LOO influence but collectively provide significant value

A valuation paradigm where data points are scored based on their impact on alignment objectives rather than raw predictive accuracy. RLHF valuation assigns higher worth to examples that steer model behavior toward desired outputs.

• Preference pairs: Comparative data where one response is preferred over another carries higher valuation weight
• Reward modeling: Data points that significantly shift the reward model's parameters receive elevated value scores
• Safety-critical valuation: Examples demonstrating refusal of harmful requests are disproportionately valuable for alignment

A method that values data based on how well it represents rare or underrepresented regions of the feature space. Distributional valuation prioritizes data points from the tails of distributions over dense, redundant clusters.

• Density-based weighting: Inversely weights data points by local density in the feature space
• Rare event amplification: Synthetic augmentation targets high-value, low-frequency examples identified through distributional analysis
• Coverage metrics: Measures how completely a dataset spans the operational input domain, with gaps indicating high-value acquisition targets

A technique that scores data points by the magnitude and direction of their gradients during training. Gradient-based valuation identifies examples that cause the largest parameter updates, signaling high informational content.

• Gradient norm scoring: Data points producing large gradient norms are considered high-value for learning
• EL2N score: The L2 norm of the error vector provides a computationally cheap proxy for data importance
• Forgetting events: Tracks how often a correctly classified example is later misclassified; frequently forgotten examples are high-value and difficult to learn

The strategic application of data valuation scores to guide synthetic data generation. High-value records identified through Shapley or influence-based methods become templates for targeted augmentation.

• Value-weighted generation: Conditional GANs and VAEs are steered to produce more synthetic samples in high-value regions of the data manifold
• Budget-constrained acquisition: Valuation scores inform optimal allocation of limited synthetic generation compute toward the most impactful data profiles
• Quality-diversity tradeoff: Balances generating diverse coverage with concentrating resources on high-marginal-contribution examples

A game-theoretic framework that assigns a fair value to each training data point by calculating its average marginal contribution to model performance across all possible subsets of the dataset.

• Computes the weighted average of a point's contribution when added to every possible coalition of other data points
• Identifies high-value outliers that disproportionately improve model accuracy
• Enables pruning of low-value or detrimental records that degrade performance
• Computationally intensive: requires Monte Carlo approximation for datasets beyond trivial sizes

A first-order approximation of data value that measures the change in model loss when a single training point is removed and the model is retrained.

• Computationally cheaper than Shapley values but ignores interactions between data points
• Effective for identifying mislabeled examples that harm model performance
• Fails to capture redundancy: two near-identical high-value points each appear low-value individually
• Commonly used as a baseline against which more sophisticated valuation methods are benchmarked

A data efficiency strategy where generative models are trained specifically on high-value records identified through valuation, amplifying their statistical signal in the training distribution.

• Focuses synthetic data generation budget on rare disease phenotypes and edge cases
• Preserves privacy by synthesizing only the most informative patient trajectories
• Reduces the data valuation feedback loop: value scores guide generation, and generated data is re-valued iteratively
• Critical for clinical domains where collecting additional real data is prohibitively expensive or ethically constrained

A gradient-based method that approximates the effect of upweighting or removing a training point on model parameters without retraining.

• Uses the Hessian of the loss function to estimate parameter changes from data perturbations
• Identifies training points responsible for specific predictions, enabling model debugging
• Scales to large models and datasets where retraining-based valuation is infeasible
• Particularly useful for detecting poisoned or adversarial examples in medical imaging datasets

Valuation techniques adapted for preference-labeled data used in aligning clinical language models with human expert judgment.

• Identifies which comparison pairs most effectively teach the reward model clinical reasoning
• Prioritizes high-disagreement examples where clinician preferences diverge from model outputs
• Enables efficient allocation of physician annotation time to the most impactful training examples
• Extends Shapley frameworks to the Bradley-Terry preference model underlying RLHF reward learning

Economic frameworks that use valuation scores to establish fair compensation for data contributors in federated healthcare networks.

• Data Shapley values serve as a basis for revenue sharing among hospitals contributing to a shared model
• Enables data acquisition strategies: purchase only records with valuation above a cost threshold
• Addresses the free-rider problem in multi-institutional research consortia
• Emerging area intersecting algorithmic fairness, privacy economics, and healthcare data governance

A game-theoretic framework that assigns a marginal contribution score to each data point by averaging its impact on model performance across all possible subsets of the training data.

• Derived from cooperative game theory's Shapley value
• Computationally intensive: requires retraining on 2^n subsets
• Efficient approximations include Truncated Monte Carlo Shapley and Gradient Shapley
• Identifies high-value records for targeted synthetic augmentation
• Also surfaces mislabeled or toxic samples that degrade performance

A first-order approximation of data importance that measures the change in model loss when a single training point is removed and the model is retrained.

• Computationally cheaper than full Shapley values
• Fails to capture interaction effects between data points
• Useful as a fast heuristic for identifying outliers
• Often used as a baseline comparison for more sophisticated valuation methods

An evaluation paradigm where a model is trained entirely on synthetic data and tested on real holdout data. The performance gap quantifies the utility of the synthetic dataset.

• Directly measures whether synthetic data can substitute for real data in downstream tasks
• Complements statistical fidelity metrics with task-specific utility
• Critical for validating that high-value records are faithfully reproduced
• Low TSTR performance indicates the generator failed to capture predictive patterns

A composite metric evaluating synthetic data across three orthogonal dimensions:

• Statistical fidelity: How closely marginal and joint distributions match the original
• Utility: Performance on downstream ML tasks (measured via TSTR)
• Privacy: Resistance to membership inference and re-identification attacks

Data valuation directly informs which records require the highest quality reproduction to preserve overall dataset utility.

A mathematical framework providing provable privacy guarantees by injecting calibrated noise into algorithms. When combined with data valuation:

• High-value records may require tighter privacy budgets due to their outsized influence
• DP-SGD clips gradients to bound individual contributions
• The privacy-utility tradeoff can be optimized by allocating more noise budget to low-value samples
• Essential for synthetic patient data released to clinical research networks

A privacy audit technique where an adversary determines whether a specific record was in the training set. Data valuation intersects here:

• High-value outliers are often more vulnerable to membership inference
• Valuation scores can guide differential privacy budget allocation
• Nearest Neighbor Adversarial Accuracy (NNAA) quantifies identifiability risk
• Synthetic data generators must ensure high-value records don't leak identifiable patterns