general family · filed under post-training
Continual pretraining
Further pretraining of an existing model; the evidence does not establish one shared data-selection or context-extension setup.
Also called continued pre-training, Continuous Pretraining, continuous pretraining (CPT).
- sources
- 3
- models
- 3
- labs adopt it
- 3
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Continual Pre-Training of Large Language Models: How to (re)warm your model? (Gupta et al., 2023) paper arxiv.orgstudies rewarming/continuing pretraining on new data
Evidence
3 spans quoted from the sources, strongest treatment first.
continuous pretraining (CPT) to equip the base model with long-context ability
Starting from a base checkpoint of DeepSeek-V3.1-Terminus, whose context length has been extended to 128K, we perform continued pre-training followed by post-training to create DeepSeek-V3.2.
we perform data quality stratification through a dedicated curation pipeline and conduct continual pre-training (CPT) on high-quality subsets.
Filed alongside
Other methods under post-training :: mid-training & continual pretraining.