Open Models Almanac
Methodsoptimizationlearning-rate schedule

specific method · filed under optimization

Linear learning-rate warmup followed by a constant rate

The learning rate increases linearly from an initial value and is then held constant.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See learning-rate schedule for the methods filed beside it.

sources
2
models
3
labs adopt it
2
strongest
used

Documented in

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 3

We reset the learning rate, warmed it up linearly to a maximum of1× 10−5, about twice the final pretraining learning rate, and then held it constant for the rest of the run.

usedoptimizationin Nemotron 3.5 SuperNVIDIA
1 more from these models

The learning rate increases linearly from zero to 5× 10−6 over 9,100 original source records and remains constant thereafter.

usedoptimizationin Nemotron 3.5 SuperNVIDIA
↑