The data was subsequently deduplicated and filtered in various ways used in the Nemotron-CC data pipeline.
1 more from this model
Deduplication was done in the same way as for Nemotron-CC.
filed under data curation
Corpus deduplication removes duplicate content from a training corpus before it is used.1 It combines exact matching across dumps with fuzzy methods like MinHash and substring-level deduplication to catch near-duplicates as well as identical documents.2
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
The Common Crawl corpus passed through exact/global (cross-dump) deduplication, MinHash fuzzy deduplication, and substring deduplication— Aleph-Alpha/Kolibri-1 · Hugging Face
exact/global (cross-dump) deduplication, MinHash fuzzy deduplication, and substring deduplication— Aleph-Alpha/Kolibri-1 · Hugging Face
Also called Deduplicate crawled training data, Deduplication, Deduplication and shuffling.
5 quotes from 5 sources, one card per model, each opening with its strongest.
The data was subsequently deduplicated and filtered in various ways used in the Nemotron-CC data pipeline.
Deduplication was done in the same way as for Nemotron-CC.
The Common Crawl corpus passed through exact/global (cross-dump) deduplication, MinHash fuzzy deduplication, and substring deduplication.
Deduplication was done in the same way as for Nemotron-CC.
The data was subsequently deduplicated and filtered in various ways