Open Models Almanac
Methodsoptimizationlearning-rate schedule

specific method · filed under optimization

Batch-size warmup

Batch-size warmup ramps the batch size up in stages over early training instead of starting at its final value.1,2 It helps because the optimal batch size grows over time: early on, a smaller batch's gradient estimate suffices, while a large early batch wastes compute.3

Trade-off The schedule trades optimization efficiency against throughput, since larger batch sizes achieve higher MFU.4,5

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. we ramp up the batch size in stages — technical report
  2. Large runs commonly ramp the batch size over early training rather than starting at its final value — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  3. The optimal batch size increases throughout training. Early in training, a larger batch size wastes compute (and tokens) because a gradient estimate from a fraction of the batch’s tokens suffices — technical report
  4. the choice of schedule trades off optimization efficiency against throughput — technical report
  5. larger batch sizes achieve higher MFU in our setting — technical report

Also called Ramping the batch size over early training, Batch-size warmup during early training, batch size scheduling strategy, Staged batch-size ramp-up, batch size ramp-up.

sources
4
models
5
labs adopt it
3
strongest
used

Documented in

Evidence

8 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 2used 3not used 3

Mistral-Large-4

Mistral AIadopted1 source

we ramp up the batch size in stages with a share of tokens that grows exponentially per stage

usedoptimizationin Mistral Large 4Mistral AI

Qwen3.8-Flash-Next

Alibaba (Qwen)discussed1 source

the first maintains the peak learning rate of the constant-batch optimum, making the ramp its only difference

evaluatedoptimizationin Qwen3.8-NextQwen Team
4 more from this model

Ramping the batch size over early training ends no better than starting at the target and costs more optimizer steps, so we do not use it

not usedunclearin Qwen3.8-Flash-NextQwen Team

Neither variant improves performance ... Consequently, we do not employ batch-size warmup in our production runs.

not usedoptimizationin Qwen3.8-NextQwen Team

a batch-size warmup that costs extra optimizer steps for no gain

not usedoptimizationin Qwen3.8-Flash-NextQwen Team

the second lowers the peak learning rate to account for the smaller batch sizes in the early stages.

evaluatedoptimizationin Qwen3.8-NextQwen Team
↑