single mention · data curation
Bilingual German-English data curation
Curates a filtered corpus combining web data, synthetic rephrasings and translations, and other curated sources in German and English.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See data sourcing for the methods filed beside it.
Filed under data curation › data sourcing
Related methods
Supported by
Methods the sources say are used for this one.
- Corpus deduplicationpart of it
The Common Crawl corpus passed through exact/global (cross-dump) deduplication, MinHash fuzzy deduplication, and substring deduplication
Aleph-Alpha/Kolibri-1 · Hugging Face · implied For English web data, keyword-based filtering additionally removed sexually explicit content
The pre-training corpus was built from extracted Common Crawl snapshots. We applied a range of curation techniques, including but not limited to:
Aleph-Alpha/Kolibri-1 · Hugging Face · impliedWe removed personally identifiable information from pre-training data using regular expressions
and 5 more
grouped into five categories (low, medium-low, medium, medium-high, high) to enable quality-aware mixing
The outputs were post-processed, shuffled, indexed, and included in the pre-training mix
Aleph-Alpha/Kolibri-1 · Hugging Face · implied- Synthetic German web-data rephrasingpart of it
German synthetic rephrasings of web data were produced with Mistral-NeMo-12B
Aleph-Alpha/Kolibri-1 · Hugging Face · implied We counteracted this by upsampling German data where downstream evaluations showed gains, and by adding translated and synthetically rephrased German data
we filtered our data based on a URL filter list
Aleph-Alpha/Kolibri-1 · Hugging Face · implied
Filed alongside
Other methods under data sourcing.