Techniquesdata curationdata filtering

specific method · filed under data curation

Filtering batched auto-generated and templated content

Filtering removes batched auto-generated and templated content from web-sourced data.

Also called Quality filtering against model collapse.

sources: 2
models: 2
lab adopt it: 1
strongest: used

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 2

Documented in

DeepSeek-V4-Flash DeepSeek-V4-Pro

Evidence

2 spans quoted from the sources, strongest treatment first.

we implement filtering strategies to remove batched auto-generated and templated content

useddata curationin DeepSeek-V4DeepSeek

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence·DeepSeek·2026-04-26

V4 implements strategies to remove batched auto-generated and templated content from web-sourced data.

useddata curationin DeepSeek-V4DeepSeek

DeepSeek V4 Guide: Engram Memory, Training Data Strategy & Release Status (2026)·Kili Technology·2026-03-23

Filed alongside

Other methods under data curation :: data filtering.

CSAM filtering Sensitive data filtering CBRN pre-training data filtering Heuristic filtering Keyword- and regex-based filtering Model-based refusal and inability filtering Pass@100 filtering Pathological repetition filtering SmolVLM-based image-text quality scoring Unified data filtering pipeline Automated verification Coding-agent session filtering and trajectory deduplication Conservative model-based noise filtering Content quality and safety filtering Continuous contribution-score ranking Data cleaning Data filtering pipeline Dense annotation of ambiguous low-quality data Domain filtering with heuristics, classifier scoring, and deduplication Evidence-based data cleaning and training constraints Filtering low-information model-generated content Heuristic and statistical filtering with deduplication and quality models Image-text relevance filtering Long-context data cleaning pipeline