specific method · filed under data curation
Domain filtering with heuristics, classifier scoring, and deduplication
Each domain is filtered using rule-based heuristics, classifier-based quality scoring, and deduplication.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
Each domain is filtered by a combination of rule-based heuristics, classifier-based quality scoring, and deduplication, with domain-specific sampling rates determined by ablation studies on smaller models.
useddata curationin Kimi K3Moonshot AI
Filed alongside
Other methods under data curation :: data filtering.
CSAM filteringSensitive data filteringCBRN pre-training data filteringFiltering batched auto-generated and templated contentHeuristic filteringKeyword- and regex-based filteringModel-based refusal and inability filteringPass@100 filteringPathological repetition filteringSmolVLM-based image-text quality scoringUnified data filtering pipelineAutomated verificationCoding-agent session filtering and trajectory deduplicationConservative model-based noise filteringContent quality and safety filteringContinuous contribution-score rankingData cleaningData filtering pipelineDense annotation of ambiguous low-quality dataEvidence-based data cleaning and training constraintsFiltering low-information model-generated contentHeuristic and statistical filtering with deduplication and quality modelsImage-text relevance filteringLong-context data cleaning pipeline