Kolibri-1
Aleph Alphasubject to the repetition caps at a 5T budget and a KL penalty towards a reference mix, which keeps the solution near a hand-set mix.
1 more from this model
The KL pull trades quality against repetition
specific method · filed under data curation
Constraining mixture optimization toward a reference or baseline mix with a KL penalty, with one described formulation also accounting for source repetition caps.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See data mixture & curriculum for the methods filed beside it.
Also called KL pull, a KL penalty towards a reference mix.
3 quotes from 2 sources, one card per model, each opening with its strongest.
subject to the repetition caps at a 5T budget and a KL penalty towards a reference mix, which keeps the solution near a hand-set mix.
The KL pull trades quality against repetition
To avoid unrealistic shifts toward a small number of dominant sources, we regularize the optimization toward the baseline prior