We reset the learning rate, warmed it up linearly to a maximum of1× 10−5, about twice the final pretraining learning rate, and then held it constant for the rest of the run.
1 more from these models
The learning rate increases linearly from zero to 5× 10−6 over 9,100 original source records and remains constant thereafter.