Open Models Almanac
Methodsoptimization

specific method · filed under optimization

Model souping

Forms a soup by taking the mean of corresponding parameters from two models.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See optimization for the methods filed beside it.

Also called soup.

sources
2
model
1
lab adopt it
1
strongest
used

Documented in

Aleph Alpha

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2

Kolibri-1

Aleph Alphaadopted2 sources

To produce the final checkpoint we merged two checkpoints trained for the same token horizon on two different data mixtures.

usedoptimizationin KolibriAleph Alpha
1 more from this model

each parameter of a soup is the mean of its values in the two models.

useddata curationin KolibriAleph Alpha
↑