Open Models Almanac
Methodsoptimizationtraining stability

implementation detail · filed under optimization

Exponential moving average (EMA)

Exponential moving average (EMA) computes an average over recent checkpoints to produce smoother weights.1 It is used to reduce variation in the final checkpoint, with the EMA weights serving as the final base checkpoint.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we apply an exponential moving average (EMA) over the 10 most recent checkpoints — laguna m1 xs2 technical report
  2. To reduce variation in the final base checkpoint, we apply an exponential moving average (EMA) over the 10 most recent checkpoints — laguna m1 xs2 technical report

Also called disabling EMA.

sources
2
models
2
labs adopt it
2
strongest
used
↑