specific method · filed under data curation
MergeMix
MergeMix is a data-mixture search method for SFT that trains one specialist model per dataset cluster instead of one model per candidate mix.1 A candidate mix is scored by averaging the specialists' parameters, each weighted by its cluster's share, and evaluating the merged model.2
Trade-off It cuts training compute for the search, since each candidate costs one evaluation instead of one training run.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
fine-tuning one specialist model per cluster
— tech reportT o score a candidate mix, we average the specialists’ parameters, each weighted by its cluster’s share in the candidate mix, and evaluate the merged model
— tech reportAfter the specialists are trained, each candidate costs one evaluation instead of one training run
— tech report
Also called MergeMix dataset-mix proposal procedure, SFT data-mix weighting with MergeMix, Mix Weighting.