specific method · filed under optimization
Linear warmup followed by cosine decay
A schedule that linearly warms up the learning rate and later decays it according to a cosine schedule.
Also called linear warmup then cosine decay.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
The learning rate is linearly warmed up over the first 2000 steps and then maintained at 2.6×10 −4 until 28T tokens. Between 28T and 40T tokens, we decay the learning rate to 2.6× 10−5 following a cosine schedule.
usedunclearin DeepSeek-V4.1-FlashDeepSeek
Filed alongside
Other methods under optimization :: learning-rate schedule.
Cosine decayWarmup-Stable-DecayBatch-size warmupTraining without batch-size warmupBatch-size warmup with adjusted peak learning rateBatch-size warmup with constant-batch optimum peak learning rateEngram learning-rate scalingFixed elevated learning rateLearning-rate annealingScaling-law fitScheduled batch-size growthWSD-specific scaling law