Mistral-Large-4
Mistral AIwe ramp up the batch size in stages with a share of tokens that grows exponentially per stage
specific method · filed under optimization
Batch-size warmup ramps the batch size up in stages over early training instead of starting at its final value.1,2 It helps because the optimal batch size grows over time: early on, a smaller batch's gradient estimate suffices, while a large early batch wastes compute.3
Trade-off The schedule trades optimization efficiency against throughput, since larger batch sizes achieve higher MFU.4,5
Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.
we ramp up the batch size in stages— technical report
Large runs commonly ramp the batch size over early training rather than starting at its final value— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
The optimal batch size increases throughout training. Early in training, a larger batch size wastes compute (and tokens) because a gradient estimate from a fraction of the batch’s tokens suffices— technical report
the choice of schedule trades off optimization efficiency against throughput— technical report
larger batch sizes achieve higher MFU in our setting— technical report
Also called Ramping the batch size over early training, Batch-size warmup during early training, batch size scheduling strategy, Staged batch-size ramp-up, batch size ramp-up.
8 quotes from 4 sources, one card per model, each opening with its strongest.
we ramp up the batch size in stages with a share of tokens that grows exponentially per stage
including the Muon optimizer, cosine decay, and batch size warmup
we also employ a batch size scheduling strategy that increases the batch size (in tokens) from a small size to 75.5M
the first maintains the peak learning rate of the constant-batch optimum, making the ramp its only difference
Ramping the batch size over early training ends no better than starting at the target and costs more optimizer steps, so we do not use it
Neither variant improves performance ... Consequently, we do not employ batch-size warmup in our production runs.
a batch-size warmup that costs extra optimizer steps for no gain
the second lowers the peak learning rate to account for the smaller batch sizes in the early stages.