Core-Set Selection

At its core, core-set selection is a geometric covering problem. The goal is to find a small subset of points whose convex hull or minimum enclosing ball approximates the shape of the full dataset. This ensures the selected subset captures the data's manifold structure and boundaries. Common algorithms, like k-Center and k-Median, iteratively select points to minimize the maximum distance (coverage radius) from any point in the full set to its nearest neighbor in the core-set.

• Key Metric: The approximation factor guarantees the core-set's solution is within a multiplicative factor of the optimal solution on the full data.
• Example: For a dataset of sensor readings, a core-set would include readings from the operational extremes and the dense central cluster.

Beyond geometry, a high-quality core-set must preserve the statistical properties of the original dataset. This means the empirical distribution of the core-set—including means, variances, and higher-order moments—should closely match that of the full data. Techniques often involve optimizing for submodular functions that maximize the diversity or coverage of the selected subset.

• Use Case: When training a model for anomaly detection, the core-set must adequately represent the 'normal' data distribution to prevent the model from learning a biased or incomplete notion of normality.
• Connection: This property is crucial for ensuring the model trained on the core-set generalizes well, as it has seen a faithful sample of the underlying data-generating process.

The primary value proposition of core-set selection is drastic reduction in training time and resource consumption. By selecting only 1-10% of the original data, training iterations become orders of magnitude faster. Algorithms are designed to be sublinear or linear in the dataset size n, making them scalable to massive datasets unsuitable for full-batch processing.

• Mechanism: Many algorithms use greedy selection or importance sampling strategies that provide provable approximation guarantees with a single pass or few passes over the data.
• Edge Relevance: This is paramount for on-device training or federated edge learning, where compute, memory, and energy are severely constrained. A compact core-set enables feasible model updates directly on the device.

Core-set selection is distinguished from simple random sampling by its foundation in computational geometry and combinatorial optimization, which provides rigorous mathematical guarantees. For many learning problems (e.g., clustering, regression, classification), it can be proven that the model loss on the core-set is within an ε-factor of the loss on the full dataset.

• Formal Definition: A set C is an ε-core-set for a function F if (1-ε)F(S) ≤ F(C) ≤ (1+ε)F(S) for the full set S.
• Implication: These guarantees provide confidence that optimization and inference performed on the tiny core-set are provably representative of the full-data solution, a critical assurance for production systems.

While both aim for data efficiency, core-set selection and active learning solve complementary problems. Active learning sequentially queries an oracle (human) to label the most informative points, minimizing labeling cost. Core-set selection identifies a representative subset from an already labeled or self-supervised dataset, minimizing computational cost.

• Synergy: Core-set methods are often used as the acquisition function within an active learning loop. The algorithm selects the batch of points that best expands the geometric coverage of the current core-set.
• Differentiation: Core-sets are typically selected independently of the model parameters, whereas some active learning strategies (like uncertainty sampling) are model-dependent.

Core-set selection is a foundational technique for the Efficient Data Strategies for Edge pillar. Its ability to distill large datasets into tiny, potent subsets directly enables several edge AI paradigms:

• Federated Edge Learning: Devices can compute model updates on a local core-set, drastically reducing communication rounds and bandwidth by sending only updates based on representative data.
• Continual Learning on Edge: As new data streams onto a device, core-set selection can be used to update a replay buffer, maintaining a compact yet representative memory of past experiences to mitigate catastrophic forgetting.
• On-Device Fine-Tuning: Before performing parameter-efficient fine-tuning (PEFT) like LoRA on a device, a core-set can be selected from user-specific data to ensure the few update steps are computationally feasible and effective.

Core-set selection is essential for on-device training and continual learning on edge hardware (e.g., smartphones, IoT sensors). By identifying a compact, representative subset of locally generated data, it enables efficient model updates without overwhelming limited memory and compute. This supports federated learning scenarios where only model updates, not raw data, are shared, preserving privacy and bandwidth.

• Key Benefit: Enables practical continual learning on edge by reducing the computational footprint of each training iteration.
• Example: A smart camera identifying new objects by training on a core-set of recent frames, not its entire video history.

When fine-tuning large foundation models (e.g., LLMs, Vision Transformers) on domain-specific data, core-set selection drastically reduces training time and cost. Instead of using millions of examples, training on a carefully chosen representative subset can achieve comparable accuracy. This is a form of data-efficient fine-tuning.

• Mechanism: Uses geometric criteria (like k-center or k-medoids) to select samples that best cover the data distribution.
• Impact: Can reduce fine-tuning dataset size by 70-90%, slashing GPU hours and cloud costs while maintaining model performance.

Core-set selection is a powerful query strategy within active learning frameworks. Instead of randomly selecting data for human labeling, it chooses the most informative and diverse points—those that form a core-set of the unlabeled pool. This maximizes the value of each expensive annotation.

• Process: Iteratively selects batches that increase the geometric coverage of the labeled set.
• Result: Achieves target model accuracy with significantly fewer labeled examples, optimizing annotation budgets. This is critical for domains like medical imaging where labeling is expert-intensive.

Core-set methods can improve model robustness by filtering out outliers and redundant samples. Algorithms can be designed to select a subset that represents the central, high-density regions of the data, inherently down-weighting noisy edge cases. This acts as a form of data pruning for quality.

• Application: In fraud detection or anomaly detection, where the majority class (normal transactions) is vast, a core-set can provide a clean, representative sample of normal behavior for training.
• Benefit: Leads to more generalizable models less likely to overfit to noise or trivial correlations in the full dataset.

Core-set selection validates and distills synthetic datasets. After generating artificial data, a core-set can be extracted to identify the most prototypical synthetic samples that best approximate the real data manifold. This creates a high-fidelity, minimal synthetic dataset for training.

• Workflow: 1. Generate a large synthetic dataset. 2. Apply core-set selection to find the most representative subset. 3. Train on this distilled set.
• Advantage: Reduces storage and training costs associated with massive synthetic datasets while preserving the essential statistical properties needed for model convergence.

Core-set selection is a precursor and inspiration for advanced data distillation (or dataset condensation) techniques. While classic core-set methods select existing data points, data distillation synthesizes new, informative samples. However, the objective is aligned: creating a minimal dataset that preserves the training dynamics of the original data.

• Comparison: Core-set selection is a selection-based approach; data distillation is a generation-based approach.
• Use Case: In research and development, core-set selection provides a strong, interpretable baseline for evaluating more complex data distillation algorithms.

Also known as dataset condensation, this technique synthesizes a small, informative dataset that yields model performance comparable to training on the original, much larger dataset. Unlike core-set selection, which selects existing points, distillation generates new synthetic samples that capture the training dynamics of the full data.

• Goal: Create a tiny, high-fidelity proxy dataset.
• Method: Often uses gradient matching or meta-learning to align the synthetic dataset's training trajectory with the original.
• Use Case: Extreme compression for rapid prototyping and hyperparameter tuning where storing the original dataset is impossible.

A machine learning paradigm where an algorithm iteratively selects the most informative data points from an unlabeled pool for human annotation. The goal is to maximize model performance with minimal labeling cost.

• Core Mechanism: Uses an acquisition function (e.g., uncertainty sampling, query-by-committee) to identify points where the model is least confident.
• Contrast with Core-Set: Active learning focuses on labeling efficiency and requires an oracle (human) to provide labels. Core-set selection assumes labels are already available and focuses on training efficiency by selecting a representative subset.
• Synergy: Core-set methods can be used as the selection strategy within an active learning loop.

The process of removing redundant, noisy, or low-quality samples from a training dataset to improve training efficiency, model generalization, and robustness. It is a broader category that includes core-set selection.

• Methods Include:
  • Forgotten Example Pruning: Removing samples the model learns quickly.
  • Gradient-Based Pruning: Eliminating samples with small gradient norms.
  • Confidence-Based Pruning: Filtering out ambiguous or mislabeled examples.
• Key Difference: While core-set selection aims for geometric representativeness, general data pruning often targets noise reduction and memorization prevention.

An approach that uses noisy, limited, or imprecise sources—such as heuristics, rules, or distant labels—to programmatically generate training data for models, reducing reliance on extensive manual annotation.

• Relation to Core-Sets: Weak supervision creates large, noisy labeled datasets. Core-set selection can then be applied to this dataset to identify a high-quality, manageable subset for efficient training or for human review in a human-in-the-loop system.
• Framework: Often implemented via data programming, where multiple labeling functions are combined.

A decentralized machine learning approach where a global model is trained across multiple edge devices or servers holding local data samples, without exchanging the raw data itself. Only model updates (e.g., gradients) are shared.

• Efficiency Challenge: Communication of model updates is the bottleneck. Core-set selection can be applied locally on each device to create a representative subset of the device's data before computing an update, drastically reducing communication costs.
• Privacy Synergy: Combining federated learning with core-set selection on the client side enhances privacy by further minimizing the exposed data footprint.

The process of updating or fine-tuning a machine learning model directly on an edge device using locally generated data, without sending raw data to a central server. This is critical for personalization and continual learning in privacy-sensitive applications.

• Core Constraint: Severe memory, compute, and power limits. Core-set selection is a foundational technique for enabling on-device training by ensuring the local training loop operates on a minimal, maximally informative dataset.
• Application Flow: 1) New data arrives on device. 2) Core-set selection merges it with existing on-device cache. 3) Model is fine-tuned on the updated core-set. This prevents unbounded growth of the training buffer.