specific method · filed under optimization
Batch-size warmup
A training strategy that ramps batch size up from a smaller value during early training.
Also called Ramping the batch size over early training, Batch-size warmup during early training.
- sources
- 2
- models
- 2
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
4 spans quoted from the sources, strongest treatment first.
including the Muon optimizer, cosine decay, and batch size warmup
Ramping the batch size over early training ends no better than starting at the target and costs more optimizer steps, so we do not use it
Neither variant improves performance ... Consequently, we do not employ batch-size warmup in our production runs.
a batch-size warmup that costs extra optimizer steps for no gain
Filed alongside
Other methods under optimization :: learning-rate schedule.