Open Models Almanac
Methodsoptimizationlearning-rate schedule

specific method · filed under optimization

Warmup-Stable-Decay

Warmup-Stable-Decay is a learning rate schedule that features a linear warmup, a stable phase at the peak, and a final decay phase.1 This schedule is attractive because a single stable-phase checkpoint can be reused across data-mix iterations via shorter cooldown runs.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. a WSD schedule: linear warmup to a peak learning rate of5×10−4, a stable phase at peak, and a final decay phase — laguna m1 xs2 technical report
  2. WSD is attractive because a single stable-phase checkpoint can be reusedacrossdata-mixiterationsviashortercooldownruns — laguna m1 xs2 technical report

Also called Warmup Stable Decay learning rate schedule, Warmup Stable Decay (WSD).

sources
4
models
3
labs adopt it
2
strongest
used

Documented in

Evidence

5 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
used 3not used 2

Kimi K3

Moonshot AInot used1 source

Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD), leading us to adopt cosine decay as the default learning rate schedule.

not usedunclearin Kimi K3Moonshot AI
1 more from this model

cosine decay consistently achieves a lower final loss than WSD.

not usedoptimizationin Kimi K3Moonshot AI

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑