Open Models Almanac
Methodsdata curationdata mixture & curriculum

single mention · data curation

Training data mixing

Training data mixing decides the proportions in which different data sources are combined into a pre-training mixture.1 It can be optimised by training small proxy models on sampled mixtures, fitting a predictive law over them, and solving a constrained program for the mix weights.2

Trade-off The KL pull trades quality against repetition, keeping small sources represented at a small cost in predicted quality.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. T o predict each benchmark’sBPB from a given data mix, we fit a log- linear law over the swarm of proxy models — tech report
  2. We then pick the mix x by solving the convex program — tech report
  3. The KL pull trades quality against repetition — tech report

Also called data mix.

Filed under data curation › data mixture & curriculum