implementation detail · filed under data curation
Evidence-based data cleaning and training constraints
Data cleaning and training constraints follow the principle of answering only when there is evidence and stating when evidence is absent.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
Hy3 implements data cleaning and training constraints based on the principle of 'answering only when there is evidence, and clearly stating when there is no evidence, to prevent confusion of information sources and data fabrication'
useddata curationin Hy3Tencent
Filed alongside
Other methods under data curation :: data filtering.
CSAM filteringSensitive data filteringCBRN pre-training data filteringFiltering batched auto-generated and templated contentHeuristic filteringKeyword- and regex-based filteringModel-based refusal and inability filteringPass@100 filteringPathological repetition filteringSmolVLM-based image-text quality scoringUnified data filtering pipelineAutomated verificationCoding-agent session filtering and trajectory deduplicationConservative model-based noise filteringContent quality and safety filteringContinuous contribution-score rankingData cleaningData filtering pipelineDense annotation of ambiguous low-quality dataDomain filtering with heuristics, classifier scoring, and deduplicationFiltering low-information model-generated contentHeuristic and statistical filtering with deduplication and quality modelsImage-text relevance filteringLong-context data cleaning pipeline