specific method · filed under optimization
Scheduled batch-size growth
A batch-size schedule that increases batch size from a smaller value to a larger target during training.
Also called batch size scheduling strategy.
- source
- 1
- models
- 2
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
we also employ a batch size scheduling strategy that increases the batch size (in tokens) from a small size to 75.5M
usedoptimizationin DeepSeek-V4DeepSeek
Filed alongside
Other methods under optimization :: learning-rate schedule.
Cosine decayWarmup-Stable-DecayBatch-size warmupTraining without batch-size warmupBatch-size warmup with adjusted peak learning rateBatch-size warmup with constant-batch optimum peak learning rateEngram learning-rate scalingFixed elevated learning rateLearning-rate annealingLinear warmup followed by cosine decayScaling-law fitWSD-specific scaling law