k-NN Evaluation

The core of k-NN evaluation is using a k-nearest neighbors classifier on the frozen feature representations. No gradient-based training occurs on the target dataset. The classifier simply assigns a label to a test sample based on the majority label among its k nearest neighbors in the training set's feature space. This directly tests the linear separability and semantic clustering of the embeddings.

The model's backbone encoder (e.g., ResNet, Vision Transformer) is kept completely frozen during evaluation. No fine-tuning or parameter updates are allowed. This isolates the assessment to the quality of the pre-trained representations themselves, preventing the evaluation from becoming a measure of the classifier's ability to adapt rather than the features' inherent usefulness.

k-NN accuracy is a strong, simple proxy for how well a linear classifier would perform on the same features. A high k-NN score indicates that classes form compact, well-separated clusters in the embedding space. This is a critical property for downstream tasks, as it suggests that simple, efficient models can be deployed on top of the features.

The choice of k (number of neighbors) is a key hyperparameter:

• Small k (e.g., 1, 5): Tests local density and fine-grained class boundaries. Sensitive to noise and outliers.
• Large k (e.g., 20, 50): Tests broader region homogeneity and is more robust to label noise. Performance often plateaus or slightly decreases as k increases if clusters are well-formed.
• Common practice is to report results for k=1 and k=20 or to sweep k and report the peak.

The similarity measure used to find neighbors is crucial. Cosine similarity (or equivalently, L2 distance on L2-normalized vectors) is almost universally used. This is because contrastive and self-supervised learning methods typically optimize representations on the unit hypersphere. Using cosine distance ensures the evaluation aligns with the training objective.

k-NN evaluation is often compared with linear evaluation protocol (training a linear classifier on frozen features).

• k-NN: Faster (no training), truly non-parametric, but can be computationally expensive at inference for large datasets.
• Linear Probing: Requires a training loop, introduces optimization hyperparameters (learning rate, epochs), but is often slightly more accurate and is the standard for final benchmark reporting. k-NN is preferred for rapid prototyping and ablation studies.

k-NN evaluation is a core component of the linear evaluation protocol, the standard benchmark for self-supervised learning (SSL). The process is methodical:

• A model is first pretrained on a large, unlabeled dataset (e.g., ImageNet) using an SSL method like SimCLR or MoCo.
• The learned encoder is frozen, and its representations (features) are extracted for a labeled downstream dataset.
• A simple k-nearest neighbor classifier is applied directly to these frozen features, without any fine-tuning.
• The top-1 or top-5 accuracy of this classifier serves as the primary metric for representation quality. This protocol isolates the encoder's capability from the effects of supervised fine-tuning, providing a pure measure of the learned features' utility.

While a linear probe (a single trained layer) is also used in evaluation, k-NN offers distinct advantages that make it a favored diagnostic tool:

• Hyperparameter Simplicity: It requires no learning rate, optimizer, or number of epochs. The main parameter is k (number of neighbors), which is often set to a standard value like 20 or 200.
• Non-Parametric Nature: As a non-parametric method, it makes no assumptions about the underlying data distribution, providing a more direct test of the feature space's geometry.
• Avoids Optimization Confounds: Results are not affected by optimization difficulties or overfitting during the probe training phase.
• Interpretability: Misclassifications can be inspected by looking at the retrieved neighbors, offering insights into feature space structure. Researchers often report both linear probe and k-NN accuracy for a comprehensive view.

k-NN evaluation is ubiquitously reported in seminal SSL papers. Its implementation is straightforward but requires efficient nearest-neighbor search:

• Feature Bank Creation: After pretraining, a feature bank is created by forwarding the entire evaluation dataset through the frozen encoder and storing the L2-normalized embeddings.
• Efficient Search: For large datasets, libraries like FAISS (Facebook AI Similarity Search) are used to perform fast approximate nearest neighbor searches in high-dimensional space.
• Standard k values: Common choices are k=20 for ImageNet-scale evaluation or k=200 for smaller datasets, as used in the original MoCo paper. The accuracy is relatively stable across a reasonable range of k.
• Reporting: Results are typically presented in tables comparing k-NN accuracy across different SSL methods (SimCLR, BYOL, MoCo v2, etc.) on benchmarks like ImageNet, CIFAR-10, or CIFAR-100.

Beyond a final accuracy number, k-NN evaluation is a powerful diagnostic for understanding the geometric properties of the learned representation space.

• Uniformity and Alignment: A high k-NN accuracy indicates good alignment (similar samples are close) and uniformity (embeddings are spread out, avoiding collapse).
• Detecting Representation Collapse: If k-NN accuracy is near random chance, it may signal a complete or partial representation collapse, where the encoder fails to produce discriminative features.
• Assessing Transfer Learning: k-NN accuracy on diverse downstream tasks (beyond the pretraining domain) helps evaluate the generalizability and transferability of the learned features.
• Monitoring Continual Learning: In continual self-supervised learning, tracking k-NN accuracy on past tasks is a direct way to measure catastrophic forgetting in the feature space.

While invaluable, k-NN evaluation has limitations that necessitate complementary assessment methods:

• Computational Cost: Performing exact nearest-neighbor search on large feature banks is expensive, though mitigated by approximate methods.
• Sensitivity to k: For very small or very large k, accuracy can fluctuate, though it's generally stable in a mid-range.
• Only Measures Linear Separability: It evaluates the feature space for a simple classifier. It does not assess how features might be organized for more complex, non-linear decision boundaries.
• Complementary Protocols: Therefore, research also uses:
  • Linear Probe: A trained linear layer.
  • Few-Shot / Semi-Supervised Evaluation: Training on a small subset of labels.
  • Transfer Learning: Full fine-tuning on a different dataset.
  • Nearest Neighbor Retrieval Tasks: Directly measuring recall@k for image or text retrieval.

The linear evaluation protocol is the most common benchmark for assessing self-supervised learned representations. A linear classifier (a single fully-connected layer) is trained on top of the frozen pretrained encoder using a labeled dataset. Its accuracy directly measures the quality and linear separability of the learned features. Unlike k-NN, it is a parametric method that requires a training phase.

• Key Difference from k-NN: Tests if features are linearly separable, not just locally consistent.
• Standard Practice: Used in seminal SSL papers (SimCLR, MoCo) as the primary reported metric.
• Computational Cost: Requires training the classifier, making it slightly more expensive than k-NN but still efficient.

Semi-supervised evaluation measures a model's ability to learn from very few labeled examples. After self-supervised pre-training on a large unlabeled dataset, a classifier is fine-tuned on a small subset (e.g., 1%, 10%) of the labeled training data. This protocol tests the sample efficiency and transferability of the representations.

• Purpose: Simulates real-world scenarios where labeled data is scarce.
• Contrast with k-NN: k-NN uses the entire labeled training set for the evaluation, but the model itself is never updated. Semi-supervised evaluation updates the model parameters via fine-tuning on the small labeled set.

Few-shot evaluation is a rigorous test of representation quality where a model must generalize from only a handful of examples per class. Common settings include 5-way 1-shot or 5-way 5-shot classification. A prototypical network or a simple logistic regression model is typically trained on the support set (the few examples) and evaluated on a query set.

• Relation to k-NN: k-NN with a very small 'k' can be seen as a simple form of few-shot inference, but dedicated few-shot protocols use more sophisticated meta-learning techniques.
• Significance: Demonstrates if SSL pre-training creates a well-structured embedding space where classes form tight clusters, enabling generalization from minimal data.

Transfer learning evaluation assesses the generality of representations by measuring performance on a downstream task different from the pre-training domain. For example, an ImageNet-pretrained model is fine-tuned on datasets like CIFAR-100, Places365, or medical images. High performance indicates the model learned general-purpose features, not just dataset-specific patterns.

• Broader Scope: Goes beyond the in-domain classification tested by k-NN or linear evaluation.
• Process: Involves full or partial (parameter-efficient) fine-tuning of the model on the new task's data.
• Key Metric: The relative improvement over training from random initialization or supervised pre-training.

Retrieval metrics evaluate representation quality for search and clustering tasks without training a classifier. Given a query embedding, the system retrieves the most similar items from a gallery. Standard metrics include:

• Mean Average Precision (mAP): Measures ranking quality across multiple recall levels.

• Recall@K: The fraction of true positive items found in the top-K retrieved results.

• Non-Parametric Nature: Like k-NN, this is a direct test of the embedding space's structure.

• Use Case: Critical for evaluating models used in content-based image retrieval, face recognition, and recommendation systems.

Centroid-based evaluation is a variant of k-NN that first computes the mean embedding (centroid) for each class in the training set. During evaluation, a test sample is classified by finding the nearest class centroid (1-NN on centroids). This method is more computationally efficient than standard k-NN for large 'k' and many classes, as it reduces the comparison set from N samples to C centroids.

• Advantage: Highly efficient for deployment in production systems for quick similarity checks.
• Limitation: Assumes classes are unimodal and well-represented by their mean, which may not hold for complex, multi-modal class distributions.