Kolibri-1
Aleph AlphaWe applied independent weight decay of 2⁻¹², approximately 0.0002441, to all decay-eligible parameters. Embedding, normalization, and expert-balancing-bias parameters were excluded from weight decay.
1 more from this model
We use decoupled weight decay for all optimiser groups except the embedding and norm scales.