Open Models Almanac
Methodsdata curationdeduplication

filed under data curation

Corpus deduplication

Corpus deduplication removes duplicate content from a training corpus before it is used.1 It combines exact matching across dumps with fuzzy methods like MinHash and substring-level deduplication to catch near-duplicates as well as identical documents.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. The Common Crawl corpus passed through exact/global (cross-dump) deduplication, MinHash fuzzy deduplication, and substring deduplication — Aleph-Alpha/Kolibri-1 · Hugging Face
  2. exact/global (cross-dump) deduplication, MinHash fuzzy deduplication, and substring deduplication — Aleph-Alpha/Kolibri-1 · Hugging Face

Also called Deduplicate crawled training data, Deduplication, Deduplication and shuffling.

sources
5
models
4
labs adopt it
2
strongest
used

Documented in

Evidence

5 quotes from 5 sources, one card per model, each opening with its strongest.

How sources treat it
used 5

The data was subsequently deduplicated and filtered in various ways used in the Nemotron-CC data pipeline.

useddata curationin Nemotron 3.5 Super VLNVIDIA
1 more from this model

Deduplication was done in the same way as for Nemotron-CC.

useddata curationin Nemotron 3 Foundation training corpusNVIDIA
↑