Warmup-Stable-Decay (WSD) learning rate schedule over a total horizon of 20 trillion tokens
1 more from this model
we use a Warmup-Stable-Decay (WSD) learning rate schedule
specific method · filed under optimization
Warmup-Stable-Decay is a learning rate schedule that features a linear warmup, a stable phase at the peak, and a final decay phase.1 This schedule is attractive because a single stable-phase checkpoint can be reused across data-mix iterations via shorter cooldown runs.2
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
a WSD schedule: linear warmup to a peak learning rate of5×10−4, a stable phase at peak, and a final decay phase— laguna m1 xs2 technical report
WSD is attractive because a single stable-phase checkpoint can be reusedacrossdata-mixiterationsviashortercooldownruns— laguna m1 xs2 technical report
Also called Warmup Stable Decay learning rate schedule, Warmup Stable Decay (WSD).
5 quotes from 4 sources, one card per model, each opening with its strongest.
Warmup-Stable-Decay (WSD) learning rate schedule over a total horizon of 20 trillion tokens
we use a Warmup-Stable-Decay (WSD) learning rate schedule
we adopted a Warmup-Stable-Decay (WSD) learning rate schedule instead of cosine.
Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD), leading us to adopt cosine decay as the default learning rate schedule.
cosine decay consistently achieves a lower final loss than WSD.
Picked by hand, not extracted: where to read more, not evidence for anything on this page.