Open Models Almanac
Methodsdata curationdata mixture & curriculum

specific method · filed under data curation

MergeMix

MergeMix is a data-mixture search method for SFT that trains one specialist model per dataset cluster instead of one model per candidate mix.1 A candidate mix is scored by averaging the specialists' parameters, each weighted by its cluster's share, and evaluating the merged model.2

Trade-off It cuts training compute for the search, since each candidate costs one evaluation instead of one training run.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. fine-tuning one specialist model per cluster — tech report
  2. T o score a candidate mix, we average the specialists’ parameters, each weighted by its cluster’s share in the candidate mix, and evaluate the merged model — tech report
  3. After the specialists are trained, each candidate costs one evaluation instead of one training run — tech report

Also called MergeMix dataset-mix proposal procedure, SFT data-mix weighting with MergeMix, Mix Weighting.

sources
2
model
1
lab adopt it
1
strongest
used

Documented in

Aleph Alpha

Evidence

4 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 4

Kolibri-1

Aleph Alphaadopted2 sources

We use MergeMix to train on90B instead of 1057B tokens, i.e., 91 % less training compute, fine-tuning one specialist model per cluster.

useddata curationin KolibriAleph Alpha
3 more from this model

we use MergeMix during which datasets are grouped into clusters and a specialist is fine-tuned on each cluster

usedunclearin KolibriAleph Alpha

The final SFT training utilises a subset of the dataset pool found by applying a procedure called MergeMix

useddata curationin KolibriAleph Alpha

We adapted MergeMix to determine the relative weighting of our SFT data mix.

useddata curationin KolibriAleph Alpha
↑