Open Models Almanac
Methodspost-trainingmid-training & continual pretraining

general family · filed under post-training

Continued pretraining

Continued pretraining further trains an existing base model checkpoint on additional data instead of training from scratch.1 It is used to equip the base model with a new capability, such as long-context ability, before post-training.2

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. Starting from a base checkpoint of DeepSeek-V3.1-Terminus, whose context length has been ex- tended to 128K, we perform continued pre-training followed by post-training — paper
  2. we performed continuous pretraining (CPT) to equip the base model with long-context ability — NVIDIA Nemotron 3 Ultra Technical Report

Also called continued pre-training, Continuous Pretraining, continuous pretraining (CPT).

sources
4
models
4
labs adopt it
4
strongest
used

Documented in

Evidence

4 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
optional 1used 3

Qwen3.5-397B-A17B

Alibaba (Qwen)adopted1 source

we perform data quality stratification through a dedicated curation pipeline and conduct continual pre-training (CPT) on high-quality subsets.

usedpost trainingin Qwen3.5-OmniQwen Team

DeepSeek-V3.2

DeepSeekadopted1 source

Starting from a base checkpoint of DeepSeek-V3.1-Terminus, whose context length has been extended to 128K, we perform continued pre-training followed by post-training to create DeepSeek-V3.2.

usedoptimizationin DeepSeek-V3.2DeepSeek
↑