Kolibri-1
Aleph AlphaWe then used the method described in MergeMix (weight-averaging of domain expert continued pre-trainings over nine domains) to find promising mixture candidates.
2 more from this model
To score a candidate mix, we average the specialists’ parameters, each weighted by its cluster’s share in the candidate mix, and evaluate the merged model.
a merge of these expert checkpoints with the weights of a candidate mix approximates an 84B-token mid-training run on that mix.