specific method · filed under optimization
Cosine decay
A learning-rate schedule that decays the rate according to a cosine curve.
Also called Cosine decay learning rate schedule, cosine learning rate schedule.
- sources
- 4
- models
- 4
- labs adopt it
- 4
- strongest
- default
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- SGDR: Stochastic Gradient Descent with Warm Restarts (Loshchilov & Hutter, 2016) paper arxiv.orgintroduces cosine annealing schedule
Evidence
5 spans quoted from the sources, strongest treatment first.
Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD) [46], leading us to adopt cosine decay as the default learning rate schedule.
Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD), leading us to adopt cosine decay as the default learning rate schedule.
using a cosine learning-rate schedule with peak learning rate 1.5×10−5
including the Muon optimizer, cosine decay, and batch size warmup
cosine decay to
Filed alongside
Other methods under optimization :: learning-rate schedule.