Open Models Almanac
Methodsdata curationdata mixture & curriculum

specific method · filed under data curation

MergeMix expert-checkpoint merging

Approximating training on a candidate mixture by weighting or merging domain-specialist checkpoints according to mixture shares.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See data mixture & curriculum for the methods filed beside it.

Also called Expert Merging.

sources
2
model
1
lab adopt it
1
strongest
used

Documented in

Aleph Alpha

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 3

Kolibri-1

Aleph Alphaadopted2 sources

We then used the method described in MergeMix (weight-averaging of domain expert continued pre-trainings over nine domains) to find promising mixture candidates.

useddata curationin KolibriAleph Alpha
2 more from this model

To score a candidate mix, we average the specialists’ parameters, each weighted by its cluster’s share in the candidate mix, and evaluate the merged model.

useddata curationin KolibriAleph Alpha

a merge of these expert checkpoints with the weights of a candidate mix approximates an 84B-token mid-training run on that mix.

usedunclearin KolibriAleph Alpha
↑