Open Models Almanac
Methodsoptimizationlearning-rate schedule

methods · optimization

learning-rate schedule

16 methods filed at this node or below it, from the sources of 16 models.

In this branch 16

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 16

Cosine decay default · 4 sources · 5 quotes
Batch-size warmup used · 4 sources · 8 quotes
Warmup-Stable-Decay used · 4 sources · 5 quotes
No batch-size warmup used · 2 sources · 2 quotes
11 from one source, or only mentioned as not used
Fixed elevated learning rate used · 1 source · 1 quote
Fixed learning rate used · 1 source · 1 quote
Learning-rate annealing used · 1 source · 1 quote
Learning-rate decay used · 1 source · 1 quote
Learning-rate tuning used · 1 source · 1 quote
Linear warmup followed by step decay used · 1 source · 1 quote
Warmup-Stable-Merge used · 1 source · 1 quote
WSD-specific scaling law used · 1 source · 1 quote
Learning-rate warmup not used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

↑