Data Contamination

The most egregious form of contamination occurs when evaluation test sets or their near-duplicates appear in the training corpus. This invalidates metrics by testing memorization rather than generalization. Detection relies on n-gram overlap analysis and embedding similarity search between the training data and canonical benchmarks like MMLU or HumanEval. A model exhibiting anomalously high scores on a specific benchmark but poor real-world performance is a primary indicator.

Training on AI-generated content (AIGC) introduces statistical artifacts that homogenize output. Key signals include:

• Low Burstiness: Uniform sentence length and structure lacking human variance.
• Low Perplexity: Text that is overly predictable to a language model.
• Tail Erosion: The disappearance of rare, fringe, or minority data points from the output distribution. This creates a self-consuming loop where models amplify artifacts from prior generations.

Contamination induces a silent distribution shift where the statistical properties of the training data diverge from the target production environment. This is measured by monitoring the KL divergence or Wasserstein distance between training and serving data. A model contaminated with synthetic text may perform deceptively well on formal benchmarks while failing catastrophically on noisy, unstructured human-generated inputs in production.

A contaminated model exhibits pathological verbatim memorization of training sequences rather than learning abstract concepts. This is tested using canary strings—unique, randomized token sequences inserted into training data. If the model can reproduce these canaries with high fidelity, it confirms memorization. This behavior correlates with inflated benchmark scores and exposes the model to copyright infringement risks.

Contamination extends to preference data. When Reinforcement Learning from Human Feedback (RLHF) relies on synthetic annotators instead of humans, the reward model overfits to superficial patterns. The model learns to exploit the proxy reward function rather than aligning with true human intent. This results in stylistically polished but factually hollow or sycophantic outputs that degrade the model's utility.

Contaminated models often display broken confidence calibration. They assign extremely high softmax probabilities to incorrect answers derived from leaked benchmark data. Expected Calibration Error (ECE) spikes, and the model becomes brittle under adversarial querying. A properly calibrated model should express uncertainty on edge cases; a contaminated one projects false certainty based on memorized patterns.