Open Models Almanac
Methodsdata curationdata filtering

general family · filed under data curation

Benchmark decontamination

Benchmark decontamination removes training documents that contain benchmark content so scores reflect ability rather than memorisation.1 It detects contamination by matching training samples against evaluation items with n-gram and semantic similarity signals, then verifying candidates, catching paraphrases that string matching misses.2

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. T o reduce the risk that a benchmark score reflects memorisation of the benchmark itself, we remove from the training corpus every document that contains text from our evaluation suite — tech report
  2. Verbatim overlapmeasures how many 𝑛-grams of the question’s text appear in the sample. It detects verbatim copies, even inside longer texts, but misses any rephrasing. Semantic overlapmeasures the embedding similarity between parts of the two texts. It effectively captures paraphrases — NVIDIA Nemotron 3.5 Super and Lightning Technical Report

Also called Benchmark-data decontamination, Decontamination, Benchmark decontamination of training data, Evaluation-contamination decontamination, decontaminate the mid-training pool, Post-training benchmark decontamination.

sources
4
models
4
labs adopt it
3
strongest
core

Documented in

Evidence

8 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
used 7core 1

Kolibri-1

Aleph Alphaadopted2 sources

we remove from the training corpus every document that contains text from our evaluation suite.

coredata curationin KolibriAleph Alpha
5 more from this model

Documents in our mid-training and long-context extension datasets were scored for contamination by n-gram overlap against our evaluation suite, and contaminated documents were dropped in full.

useddata curationin KolibriAleph Alpha

we therefore decontaminate the mid-training pool, from which long-context extension also draws, and remove every document that contains text from the 226 tasks of our evaluation suite, with all their splits.

usedunclearin KolibriAleph Alpha

To avoid this, we scan the SFT data pool against our benchmarks and drop the rows that reuse benchmark text.

useddata curationin KolibriAleph Alpha

All samples undergo the same processing pipeline, which removes overlap with our evaluation benchmarks

usedunclearin KolibriAleph Alpha

We therefore decontaminate the mid- and post-training pool

useddata curationin KolibriAleph Alpha
↑