continuous pretraining (CPT) to equip the base model with long-context ability
general family · filed under post-training
Continued pretraining
Continued pretraining further trains an existing base model checkpoint on additional data instead of training from scratch.1 It is used to equip the base model with a new capability, such as long-context ability, before post-training.2
How this definition was written
Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.
Starting from a base checkpoint of DeepSeek-V3.1-Terminus, whose context length has been ex- tended to 128K, we perform continued pre-training followed by post-training
— paperwe performed continuous pretraining (CPT) to equip the base model with long-context ability
— NVIDIA Nemotron 3 Ultra Technical Report
Also called continued pre-training, Continuous Pretraining, continuous pretraining (CPT).
Documented in
Evidence
4 quotes from 4 sources, one card per model, each opening with its strongest.
Qwen3.5-397B-A17B
Alibaba (Qwen)we perform data quality stratification through a dedicated curation pipeline and conduct continual pre-training (CPT) on high-quality subsets.
DeepSeek-V3.2
DeepSeekLFM3-1.7B
Liquid AIa starting point for continued pre-training and your own post-training