specific method · filed under optimization
AdamW
The AdamW optimizer, used here for normalization weights and other non-matrix parameters.
Also called AdamW optimizer.
- sources
- 3
- models
- 4
- labs adopt it
- 2
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Decoupled Weight Decay Regularization (Loshchilov & Hutter, 2017) paper arxiv.orgintroduces AdamW
Evidence
4 spans quoted from the sources, strongest treatment first.
We maintain the AdamW optimizer for the embedding module, the prediction head module, the static biases and gating factors of mHC modules, and the weights of all RMSNorm modules
use AdamW optimizer (Loshchilov and Hutter, 2019) for the weights of all RMSNorm modules and other non-matrix parameters
We retain AdamW (Loshchilov and Hutter, 2019) for normalization-layer weights and other non-matrix parameters, including biases and scaling factors.
We employ the AdamW optimizer
Filed alongside
Other methods under optimization :: optimizer.