Open Models Almanac
Methodsdata curationdata mixture & curriculum

specific method · filed under data curation

KL-regularized data-mixture optimization

Constraining mixture optimization toward a reference or baseline mix with a KL penalty, with one described formulation also accounting for source repetition caps.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See data mixture & curriculum for the methods filed beside it.

Also called KL pull, a KL penalty towards a reference mix.

sources
2
models
2
labs adopt it
2
strongest
used

Documented in

Aleph Alpha
Poolside

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 3

Kolibri-1

Aleph Alphaadopted1 source

subject to the repetition caps at a 5T budget and a KL penalty towards a reference mix, which keeps the solution near a hand-set mix.

usedunclearin KolibriAleph Alpha
1 more from this model

The KL pull trades quality against repetition

usedunclearin KolibriAleph Alpha
↑