single mention · data curation
Training data mixing
Training data mixing decides the proportions in which different data sources are combined into a pre-training mixture.1 It can be optimised by training small proxy models on sampled mixtures, fitting a predictive law over them, and solving a constrained program for the mix weights.2
Trade-off The KL pull trades quality against repetition, keeping small sources represented at a small cost in predicted quality.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
T o predict each benchmark’sBPB from a given data mix, we fit a log- linear law over the swarm of proxy models
— tech reportWe then pick the mix x by solving the convex program
— tech reportThe KL pull trades quality against repetition
— tech report
Also called data mix.
Filed under data curation › data mixture & curriculum