Open Models Almanac
Methodsoptimizationoptimizer

specific method · filed under optimization

Decoupled weight decay

Weight decay is applied separately from the gradient update, with some parameter categories excluded in the cited configurations.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See optimizer for the methods filed beside it.

Also called independent weight decay.

sources
2
model
1
lab adopt it
1
strongest
used

Documented in

Aleph Alpha

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2

Kolibri-1

Aleph Alphaadopted2 sources

We applied independent weight decay of 2⁻¹², approximately 0.0002441, to all decay-eligible parameters. Embedding, normalization, and expert-balancing-bias parameters were excluded from weight decay.

usedoptimizationin KolibriAleph Alpha
1 more from this model

We use decoupled weight decay for all optimiser groups except the embedding and norm scales.

usedunclearin KolibriAleph Alpha
↑