Open Models Almanac
Methodsoptimizationlearning-rate schedule

specific method · filed under optimization

Cosine decay

A learning-rate schedule that decays the learning rate according to a cosine curve.

The sources on this page name it without explaining how it works. The line above is the extraction's one-line gloss. See learning-rate schedule for the methods filed beside it.

Also called Cosine decay learning rate schedule, cosine learning rate schedule.

sources
4
models
4
labs adopt it
4
strongest
default

Documented in

Moonshot AI
Xiaomi
Z.ai (Zhipu AI)

Evidence

5 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
used 3default 2

Kimi K3

Moonshot AIadopted1 source

Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD) [46], leading us to adopt cosine decay as the default learning rate schedule.

defaultoptimizationin Kimi K3Moonshot AI
1 more from this model

Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD), leading us to adopt cosine decay as the default learning rate schedule.

defaultunclearin Kimi K3Moonshot AI

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑