Data contamination
In short
Data contamination is when text from a test, such as exam questions or their answers, is already in a model's training data, so a high score can reflect memory rather than skill on new questions.
- Also called
- Benchmark contamination, Test-set contamination
- Topic
- How models are trained
- Level
- Some background helps
- First described
- Brown and colleagues at OpenAI, May 2020, in the GPT-3 paper, which names "data contamination" and measures it across their benchmarks. They note earlier overlap checks, including on GPT-2, so the worry predates that paper1.
Brown and colleagues studied what they called data contamination in the GPT-3 paper in 2020. The training text was from the web, so tests could be in it. A bug meant their attempt to strip overlaps was only partial, and retraining was too costly, so they measured the rest. They flagged an example when a 13-word sequence from it was in the training text, or the whole example if shorter1.
On a quarter of benchmarks, more than half the examples were flagged. Cleaned subsets usually scored about the same, and overlap did not track the change. They concluded they had either over-flagged, or contamination often mattered little. PIQA, a physical-commonsense test, was an exception: the clean subset scored about 3 percentage points lower1.
Dodge and colleagues found the same issue in C4, a cleaned web corpus of more than 156 billion tokens. On several writing tasks, 1.87% to 24.88% of the target texts were already in C4, including 15.49% of XSum summaries, which a model could copy instead of writing. On GLUE, test inputs found in C4 ranged from under 2% to over 50%. If labels are in the corpus, they say the split is no longer a fair test2.
Copying is not guaranteed. Magar and Schwartz pre-trained BERT with labelled test items mixed in, then fine-tuned. Sometimes the model scored higher on items it had seen. Sometimes it had memorised them but the task score did not rise. Repeats and model size moved memorisation and this exploitation differently3.
Llama 3 shows the same mix. Using 8-word overlaps, and a cutoff chosen to maximise an estimated score gap, Meta put about 85% of HellaSwag in the contaminated set, with an estimated lift of about 14 to 15 points, against about 52% of Natural Questions and a lift near 1 point. On MMLU the overlap was too high to estimate a lift. They note every detector has false alarms and misses4.
Example
In the public web corpus C4, 15.49% of the target summaries in the XSum news test were already present word for word, so a model could copy them instead of writing a summary2.
Why it matters
A leaderboard score is evidence of skill only if the test was unseen. Contamination makes a familiar exam look like a new one, and the size of the cheat is different for every test.
Questions people ask
Is data contamination the same as overfitting?
No. Overfitting is fitting the training set too tightly. Contamination is a test question, or its answer, turning up inside that training set, so the test is no longer unseen. Brown and colleagues treat it as a separate measurement problem from ordinary train-versus-validation overfitting1.
Does contamination always inflate the score?
No. On GPT-3, most cleaned subsets scored about the same as the full test1. Llama 3's estimates showed a large possible lift on HellaSwag and almost none on Natural Questions, despite heavy overlap on both4. Magar and Schwartz also found cases of memorising a leaked item without a higher task score3.
Related terms
Sources
- Brown et al. (2020), Language Models are Few-Shot Learners, arXiv:2005.14165, section 4
- Dodge et al. (2021), Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus, arXiv:2104.08758
- Magar and Schwartz (2022), Data Contamination: From Memorization to Exploitation, ACL, arXiv:2203.08242
- Grattafiori et al. (2024), The Llama 3 Herd of Models, section 5.1.4, arXiv:2407.21783