specific method · filed under optimization
Warmup-Stable-Decay
A learning-rate schedule identified as WSD, distinct in the evidence from cosine decay.
Also called Warmup Stable Decay learning rate schedule, Warmup Stable Decay (WSD).
- sources
- 4
- models
- 3
- labs adopt it
- 2
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (Hu et al., 2024) paper arxiv.orgintroduces the WSD learning-rate schedule
Evidence
5 spans quoted from the sources, strongest treatment first.
Warmup-Stable-Decay (WSD) learning rate schedule over a total horizon of 20 trillion tokens
we adopted a Warmup-Stable-Decay (WSD) learning rate schedule instead of cosine.
we use a Warmup-Stable-Decay (WSD) learning rate schedule
cosine decay consistently achieves a lower final loss than WSD.
Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD), leading us to adopt cosine decay as the default learning rate schedule.
Filed alongside
Other methods under optimization :: learning-rate schedule.