Open Models Almanac

Data · D2

Data mixture and curriculum: choosing what to read, and when

Fitting a data mix to an objective makes gains according to that objective, but risks losing what you don't measure, and is still affected by your chosen prior

Illustration of a row of glass jars of coloured sand in very different sizes, a rack of small vials each layered differently, and one tall glass holding a wide dark lower layer and a smaller, brighter upper layer.

Kolibri, Aleph Alpha’s 78B-parameter model, reads 20T tokens in its main stage, drawn from 78 datasets that hold about 29.3T tokens between them. Someone has to decide how many of those 20T come from each dataset. That’s 77 parameters to the data mix (the last share is whatever remains), and you only want to train once, rather than doing multiple full scale runs and keeping the best.

Some labs change the mix between stages, keeping broad web text for early training and spending scarcer, better data late. A few also fit the proportions, by training a swarm of small models on different mixes and solving for the mix a fitted law says is best. Two of the 16 labs on this site, Aleph Alpha and Poolside, publish a method for fitting.

A solver with no penalty gives four of Kolibri’s nine mid-training domains exactly no tokens. To prevent that, Aleph Alpha adds a penalty toward a mix people chose in advance. The fit then gives up what its objective weights lightly, and part of what comes out is that prior.

Background

A mix gives each data source a share of a stage’s tokens. Sources group into domains such as web text, code or math. A run’s budget is the number of tokens it reads, counting repeats, and those are tokens seen. A source’s pool is what it holds, and unique tokens are the distinct ones actually read. A share of a budget divided by the pool gives passes: 4T seen from a 500B source is 8 passes, and epochs mean the same thing.

Runs come in stages. Pre-training is the long first stage on broad text. Mid-training is a shorter base-model stage on better data, and long-context extension lengthens the sequences. Supervised fine-tuning (SFT) then trains on conversations and turns a base model into an assistant. Scores taken after a short test SFT are closer to what ships, so reports use them to judge a base mix. Each lab draws these lines its own way.

As in the primer, a proxy is a small model standing in for a run nobody can repeat, and an ablation changes one piece of a recipe to see what it did. A checkpoint average combines the weights from a run’s last few saves, to smooth out step-to-step noise. M4 means four saves. A wider window scores higher by itself, so only same-size averages compare. The reports call these merges, and we keep that word for something else below.

Bits per byte (BPB) is cross-entropy loss per byte of benchmark text, where lower is better. Accuracy benchmarks count right answers: GSM8K is math word problems, HumanEval is small coding tasks, MMLU and TriviaQA test knowledge, and HellaSwag and ARC test commonsense reasoning. Kolibri’s aggregate score averages knowledge, math and code in each language, then weights English 80 and German 20. The report’s knowledge group includes HellaSwag and ARC. Its SFT score is a different number: the mean over 11 benchmarks after one fixed test SFT. Repeating that test SFT from the same base with a new seed gives a different score, and that spread is the eval noise the report reads differences against.

Modeling code records none of this, so we read technical reports first, then cards and official blogs. Kolibri’s report is the most detailed public account and our worked example, and its choices may not be typical. Reports describe what shipped, and only two of 16, Poolside and Aleph Alpha, tell of a mix or a mixing method that fell short.

Figure 1.Aleph Alpha's process for Kolibri, one model's sizes in one place. Only the top lane is the shipped model. Lanes below it aren't to scale.

Later stages get different data

Many labs change data between stages with no optimiser at all. Of 16 labs, six state that a later base-model stage uses different data (NVIDIA, Mistral, Aleph Alpha, Liquid, Xiaomi, Z.ai). Four more say something partial (Poolside, Moonshot, DeepSeek, Qwen). We found nothing on it from the other six, which doesn’t mean they don’t do it.

High-quality data is small, so a lab can’t put a whole run’s budget on it without repeating it many times. Hu et al. (MiniCPM, 2024) mixed instruction-style and high-quality data into only the last stretch, the learning-rate decay of about 10%, and scored models after an identical SFT. On a 2.4B model, MMLU rose from 44.6 to 50.9 and GSM8K from 27.7 to 42.3. On a 1.2B model most scores rose too, but GSM8K fell from 34.2 to 31.8. No seeds or intervals are given. A longer SFT alone (12B against 6B tokens) moved almost nothing, so they rule out a shortage of SFT tokens and recommend that specialisation “should start from the decay phase”.

Feng et al. (2024) made this a recipe with a controlled comparison. On an 8B model at 1T tokens, the same source weights in two phases beat the same weights in random order: an average score of 52.86 against 51.12, which is 1.74 points, or the paper’s relative “3.4%”. The gain is mostly GSM8K (+9.47) and code (+2.78), and MMLU slightly drops. Among the lengths tried, 40% in the second phase scores best. They report no error bars, and their 15T run has no baseline at 15T.

Scarcity explains why good data gets a small share, and doesn’t explain why it goes last, since Feng’s comparison changes only the order. One picture helps (ours, from standard SGD on a quadratic, and networks aren’t quadratic). In a direction of curvature hh, each later step shrinks an earlier step’s contribution by a factor 1−ηh1 - \eta h, with η\eta the learning rate. Late tokens therefore count for more in the final weights, and early ones fade, which is the benefit and the cost in one effect.

Labs split at very different points. Nemotron 3 Ultra goes about 75/25 inside pre-training (15T then 5T), and Nemotron 3 Nano switches at 94%. Mistral Large 4 has 65/35 by design, in a report whose Figure 8 caption describes a run at 8×10²³ FLOPs that restarted from the end of phase 1 and “trained for approximately two-thirds of the original phase 2”. Kolibri’s separate stage comes to about 85/15 (3.44T after 19.99T, computed). NVIDIA cites Feng, Aleph Alpha cites Hu, and Mistral says “two-phase curriculum” and cites neither, so we can’t trace adoption from one lab to the next. We found no lab reporting a test of its split point at full scale.

Figure 2.Shares of tokens in each phase, from each report's own figure. Taking half the sum of absolute changes, Mistral moves 24.3 points and NVIDIA 15.7 (computed). Kolibri's domains move 64.2 points from pre-training to mid-training by the same measure (computed from its card).

Labs use the word for different changes. Mistral moves whole domains, code from 16.9% to 39.0% and web from 56.3% to 32.0%. NVIDIA keeps code at 14.0% and swaps tiers, dropping medium crawl and adding high-quality PDFs. Bengio et al. (2009) defined a curriculum as a sequence of reweightings whose weights only grow until the training distribution is reached. Tiers dropping to zero, as at NVIDIA, don’t fit that (our reading). Mistral’s shift can fit it if phase 2 is taken as the target, since a share can fall while its weight doesn’t. So “curriculum” here is loose usage for a mix that changes over time.

Better data late doesn’t hold at every step. Kolibri’s long-context stage swings back toward documents, with 15.8% English web and 33.3% OCR’d PDFs (its card). The stage takes a third of its tokens from a pool of naturally long documents and reweights part of the rest toward length, and “long documents are concentrated in web text and scanned material”. The report calls the resulting shift between domains “a consequence of the length reweighting rather than an explicit design choice”.

No source cleanly measures what a shift costs, and sequence length often changes with the mix. Kolibri goes from 16k to 64k at its switch, Mistral from 4K to 40K, and Liquid and Z.ai extend context in the same stage. Their results can’t separate the two.

Fitting a mix on a swarm

Poolside writes that “optimizing data mixtures manually becomes intractable as the number of datasets, capability trade-offs, and training constraints grows”. On M.1, the model before XS.2, a manual mix gave “excessive repetition in high-value subsets” and “suboptimal allocation of data budget across different sources”.

The published methods follow one schema, named by Olmix (Chen et al., 2026). Train a swarm of proxies on different mixes. Fit a regression from mix to score. Solve for the mix the regression says is best. Mixes are drawn from a Dirichlet distribution, a distribution over shares that sum to one, with a concentration setting how far they stray from a centre. RegMix (Liu et al., 2024) showed that a regression from 1M models can rank mixes on 1B models: rank correlation 0.97 over 64 mixes, on validation loss on one web source, using gradient-boosted trees.

Aleph Alpha’s pre-training search trains 3,354 proxies of 30M parameters, dense with no experts, on 3B tokens each, centred on natural shares (proportional to source size). For each of 24 benchmarks it fits a law of the mix xx,

BPBt(x)=eat+ebt⋅x\mathrm{BPB}_t(x) = e^{a_t} + e^{b_t \cdot x}

where ata_t gives a floor and btb_t holds one coefficient per source. Subtract the floor and take logs, and the result is linear in xx, hence log-linear. The form comes from Ye et al. (2024), who call their laws “empirical findings”, and it reaches Kolibri through Olmix. The report prints no goodness of fit. It then solves

min⁡x  ∑twt ebt⋅x+λ DKL(x ∥ q),xi≥0,    ∑ixi=1,    xi≤κi\min_x \; \sum_t w_t\, e^{b_t \cdot x} + \lambda\, D_{\mathrm{KL}}(x \,\|\, q), \quad x_i \ge 0,\;\; \sum_i x_i = 1,\;\; x_i \le \kappa_i

with equal weights wtw_t, a prior qq, a Kullback-Leibler (KL) penalty that grows as the mix leaves qq, and caps κi\kappa_i that limit repetition. The floor is constant and drops out. Each term is convex and the constraints are linear, so a standard solver finds the optimum, and the hard part is the noise in what it’s optimising. (aa, bb, ww, tt follow the report and are local to this section.)

The estimate has separate sources of error. The gap between proxy and target is a bias, as is any mismatch between the true response and the law’s form, and more runs fix neither. Noise in the coefficients is variance, which more runs do reduce. Solving over a noisy fit adds an optimistic bias, since the solver picks whichever coefficients happen to look best, so the fit’s own predicted score flatters its choice.

Aleph Alpha’s swarm comes to 10T tokens (computed), which reads like half a run. By tokens times parameters it’s about 0.02% of one (crude, across dense and MoE). The limit is many small jobs and many evaluations. The gap to the target is larger. These models are 2,600 times smaller by total parameters and dense, while Kolibri is a mixture-of-experts (MoE) hybrid, so they can’t show anything a mix does through expert routing. Olmix tested this proxy size against 1B targets (rank correlation 0.896), which stops far short of 78B. Olmix also finds 1M proxies unreliable (r=0.73r = 0.73) and recommends at least 15M, against RegMix’s 1M, while noting that RegMix’s public code is closer to 15M.

The report warns about its own fit: the optimiser “is sensitive to the number of swarm samples, and the prior and its two knobs (λ, T) influence the predicted optimum dramatically”. It also writes that “smaller sources are typically less well-explored even at the lowest concentrations”, and a Dirichlet draw’s variance in a share, m(1−m)/(c+1)m(1-m)/(c+1) for a centre share mm and concentration cc, does get smaller in absolute terms as the share does. It had to patch token-count errors, and “the sampling prior for the swarm ablations, computed from the old counts, cannot be corrected after the fact”. The fit also doesn’t follow the filter: “some of our German quality-classifier buckets are down-weighted relative to their labels”.

The report prints no benchmark gain for this mix against any other. It gives a predicted BPB, 3.670 ± 0.002 against 3.667 ± 0.002 for the unconstrained optimum, from the same fit that chose the mix. The card says predictions were “confirmed using a separate validation proxy setup”, with no numbers. At domain level the mix ends near the pool’s shares: English web 43.4% against 41.8%, code 13.6% against 14.8%. Specialized domains (legal and public-sector text) have four times their pool share, close to their cap, and STEM QA two thirds of its. That needn’t mean the fit did little, since its prior isn’t the pool, and neither the prior nor the per-source weights are printed at that level.

Poolside’s AutoMixer has the same shape on another scale: about 60 proxies per ablation, 0.5B parameters MoE on 60B tokens, over more than 50 dataset groups, one non-linear surrogate per capability group, in a form it doesn’t state. Olmix’s minimum for a log-linear law would be over 150 runs there, but that rule needn’t transfer, since Poolside’s search stays inside an L1 ball around its prior, both when sampling and when solving. That keeps the search local, and it avoids fitting in one place and solving in another.

Merging experts in place of runs

A fresh tiny model may not test a mid-training mix. Writing of RegMix’s 1M proxies, Wang et al. (2026) say “such tiny proxies are ineffective for capturing the emergent capabilities” that stage needs. A proper test starts from a trained base, and doing that for every candidate is expensive. MergeMix estimates each candidate with no training on the mix.

From one base Θ0\Theta_0, train one expert per domain on that domain alone. These are whole models, unrelated to the experts inside an MoE layer. For a candidate mix xx, evaluate an expert merge,

Θmerge(x)=Θ0+∑kxk(Θk−Θ0).\Theta_{\text{merge}}(x) = \Theta_0 + \sum_k x_k (\Theta_k - \Theta_0).

Here is why that can work. Write ∇k\nabla_k for domain kk‘s gradient at Θ0\Theta_0. After nn steps at learning rate η\eta, an expert has moved about −ηn∇k-\eta n \nabla_k, so the merge has moved −ηn∑kxk∇k-\eta n \sum_k x_k \nabla_k. The gradient of a mixed loss is the mix of the gradients, so a run on mixed data for nn steps moves the same way, to first order in ηn\eta n. At second order, with Hessians HkH_k, Wang’s terms combine into one line (our rearrangement):

Θmix−Θmerge≈−12(ηn)2 Covx(Hk,∇k)\Theta_{\text{mix}} - \Theta_{\text{merge}} \approx -\tfrac12 (\eta n)^2 \, \mathrm{Cov}_x(H_k, \nabla_k)

The covariance is taken across domains with weights xkx_k. It vanishes if the domains share a Hessian or share a gradient, so what matters is how curvature and gradient vary together. Approximation needs ηn\eta n times the curvature to be small, along with the shared start and constant learning rate that the paper lists. A shorter horizon reduces this error and leaves less signal over minibatch noise, and merging also averages the experts’ noise, which a single run doesn’t do.

This is a motivation, and not a bound. It also compares a merge with a run of the same length, and neither published use has that: Aleph Alpha’s 20B experts stand in for 84B runs, and Wang’s 5B experts for 200B runs. That the ordering of mixes survives the longer horizon is an empirical claim.

Merged point to the mixed run of the same length: 0.037.

Merged point to the mixed run 4.2× as long: 1.702.

ηn = 0.30: 600 steps at η = 0.0005.

Coral and teal: the expert paths and contours of A and B. Dotted: between the experts' ends, with the merged point (diamond) at share x. The third solid path and its filled dot: the mixed run. Dashed, open circle: the longer mixed run. Thin lines from the diamond are the two gaps.

Figure 3.A toy with two invented losses. At a short horizon the merged point and a mixed run of the same length agree, and the gap grows about with the square of the horizon until the paths near their minima. With shared curvature the gap is zero at any horizon. Published uses compare merges with longer runs, which the arithmetic doesn't cover.

Wang’s evidence: over 12 mixes in four domains on an 8B MoE model, merges rank mixes as real 200B mid-training runs do, with a Spearman correlation of 0.92 on the overall score (95% interval about 0.67 to 0.98, computed). By capability it runs from 0.72 to 0.98, with 12 mixes too few to resolve those differences. Their chosen mix beats a production manual mix by 0.5 points on the average, with no interval. Their 100× saving counts training tokens against ten manual runs, and leaves out evaluating 40 merges.

Aleph Alpha uses nine experts of 20B each, 180B in all, to stand in for 84B runs on 113 weightings. Those would cost 9,492B as real runs, a saving of 98% before validation (computed). It fits one log-linear law per benchmark to the merge scores. Then 18 real 84B runs and four 400B runs follow, which is how these mixes were actually judged. The report’s “above 0.9” is Wang’s number, and it prints no correlation of its own for mid-training merges.

Its one comparison of merges and runs is on fine-tuning, which Wang didn’t study. An SFT mix is a mix too, over clusters of conversation data. With two or three specialists (experts for one cluster) of 20B, merges and runs agree on two benchmarks, at Spearman 0.77 to 0.94 over four or six weightings. On MATH-500, all scores sit between 84 and 91 and “neither ordering carries information”. Aleph Alpha then applied the method to the whole SFT mix: 20 clusters at 4.5B each, 90B of training instead of 63 runs totalling 1,057B, “91 % less training compute” (about 79% if the eight trained candidates also ran 16.78B each, computed). “With 20 clusters, the merges find good candidates but do not rank them.”

Figure 4.Aleph Alpha's search over Kolibri's SFT mix, on the full model, one run per weighting, on its own benchmark set; these scores aren't on the scale of the proxy scores elsewhere in this article. The report can't say whether the number of specialists or their shorter training breaks the ranking.

Six ranks are few: at n=6n = 6 a correlation needs to reach 0.89 to clear the usual 5% test (computed). So −0.09 tells us this ranking wasn’t near Wang’s 0.92, and can’t separate zero from moderate agreement. The search also moved toward high merge scores, and refinements centred there could select for the proxy’s errors (our reading). The report says “our experiments do not show which of these two differences breaks the ranking”, and the differences are 20 specialists instead of two or three, and 4.5B of training instead of 20B. Aleph Alpha therefore trained eight weightings and chose on trained scores. Six of the seven besides the baseline train above it, up to 56.7 against 51.5, in one run each. That’s weak on its own: if all eight were equal, at least six would beat the baseline one time in four (computed). The best was a random draw from mid-ranking, so we can’t credit the merge over the sampling.

Here are the searches side by side, for base-model mixes.

Kolibri pre-trainingKolibri mid-trainingLaguna XS.2 AutoMixer
Proxy30M dense, 3B tokens11.9B total, 0.65B active MoE, from a 2.79T checkpointabout 0.5B MoE (total or active unstated), about 60B tokens
Runs3,3549 experts of 20B, 113 merges, then 18 runs at 84B and 4 at 400Babout 60 per ablation; number of ablations unstated
Dimensions78 sources9 domains; datasets inside a domain by a fixed rulemore than 50 dataset groups
Metricmean BPB, 24 benchmarks (18 English, 6 German)aggregate benchmark scorebenchmark evals in 5 capability groups
Lawlog-linear, per benchmarklog-linear, per benchmarknon-linear surrogate per group, form unstated
PenaltyKL to a flattened size prior, λ=0.05\lambda = 0.05KL to a hand mix, λ=0.01\lambda = 0.01KL to a manual prior, plus an L1 ball; values unpublished
Repetitionat most 3 passes at 20T, per sourceat most 4 passes across both stages, per datasetno number published

Two other labs mention proxies for mixes with no method. Moonshot sets “domain-specific sampling rates determined by ablation studies on smaller models”, and DeepSeek’s V4.1 report designs “a scaling ladder over model parameters and training data”, without saying that it sets any weight. Mistral uses that phrase for its recipe and not its mix. NVIDIA tests one dimension at a time, such as an Indic share at 10, 20 and 30%, and says these runs “inform mixture design but do not establish an optimal share for Super”. Olmix and MergeMix are external work that Aleph Alpha adopts, and Poolside cites Olmix and RegMix among its inspirations.

How the penalty and the prior set the mix

The penalty is DKL(x∥q)=∑ixiln⁡(xi/qi)D_{\mathrm{KL}}(x \| q) = \sum_i x_i \ln (x_i / q_i), which is zero at x=qx = q. In Kolibri’s pre-training solve the prior is qi∝tokensi1/3q_i \propto \text{tokens}_i^{1/3} (the report’s temperature T=3T = 3, unrelated to token budgets). Larger sources get more, though much less than in proportion: a source 1,000 times larger gets 10 times the prior share. Note that the swarm samples around natural shares, while the solve pulls toward this flatter prior. “The KL pull trades quality against repetition: at λ = 0 the solver ignores source sizes and picks only the best-fitting sources, while a large λ reproduces the size prior.”

The penalty’s effect has an exact form (our derivation, not in the report). First take the objective’s slope as fixed at sis_i for source ii. A multiplier ν\nu for ∑ixi=1\sum_i x_i = 1 gives si+λ(ln⁡(xi/qi)+1)+ν=0s_i + \lambda(\ln(x_i/q_i) + 1) + \nu = 0, so

xi∝qi e−si/λx_i \propto q_i \, e^{-s_i / \lambda}

The penalty tilts the prior exponentially toward sources with a good slope. On the real objective the slopes depend on the mix, and caps apply, so the solution satisfies the same form as a fixed point:

xi=min⁡(κi,  c qi e−si(x)/λ),si(x)=∑twt bt,i ebt⋅xx_i = \min\bigl(\kappa_i,\; c\, q_i\, e^{-s_i(x)/\lambda}\bigr), \quad s_i(x) = \sum_t w_t\, b_{t,i}\, e^{b_t \cdot x}

with cc set by ∑ixi=1\sum_i x_i = 1. Three things follow. For any λ>0\lambda > 0 no share is exactly zero, though it can be tiny: with λ=0.01\lambda = 0.01, a slope gap of 0.2 gives a factor near 10−910^{-9}. At λ=0\lambda = 0, a single benchmark with no caps would put everything in one source, since a linear function on shares is lowest at a corner. With caps it fills the best sources in order. With several benchmarks, slopes equalise across the sources in use below their caps, so several can survive.

The third follows from counting. The first term sees the 78 shares only through 24 numbers, bt⋅xb_t \cdot x. So at least 53 directions in the mix leave the fit unchanged (computed), and along them only the prior and the caps decide. Among the mixes that score the same, the solver returns the one closest to the prior in KL that fits under the caps. This form is old: statisticians call it exponential tilting, or the Esscher transform (Esscher, 1932), and one step of it is exponentiated gradient (Kivinen and Warmuth, 1997).

Mid-training gives a printed series. Its mixes need names, since the report’s labels appear in the tables.

NameWhat it isPrior, λ\lambda
C1, C2Hand mixes with identical domain shares, differing in how datasets are sampled inside a domain
C3Hand mix with more instruction, STEM and agentic code, chosen from 400B proxy runs
S1, S2, S4Solves against C2λ=\lambda = 0, 0.01, 0.05
S3Near-twin of S2, projected from a 5T budget, as S1 and S4 areC2, 0.01
S5Solve against C3C3, 0.01
ProductionS5 with German instruction raised by hand to its cap, then trained on

The report prints neither the law nor the penalty for mid-training, and that solve maximises a score, so the formulas above carry over only if the form does. A mid-training mix also has a second axis the solve leaves alone: datasets within a domain follow a fixed rule, since optimising them “requires one expert per dataset, 87 trainings in total”.

λ = 0.01 (S3, projected). These steps are all that's printed.

At or near the computed 5T cap: English instruction, German instruction. Outline: hand mix C2. Triangle: the cap at 5T under a 30T pre-training, computed, for the domains where it may set the share (our inference); filled when at or near it.

All five solutions, as a table
%C2S4S3S2S1
English web33.020.33.13.10.0
German web19.02.30.40.40.0
English instruction10.014.930.731.531.0
German instruction1.01.41.41.41.4
Code10.027.129.829.442.1
Agentic code8.59.77.97.80.0
STEM documents6.513.710.910.80.3
STEM QA9.09.615.715.525.1
Specialized3.01.00.10.10.0
M1060.961.162.962.461.8
SFT40.639.942.040.840.9
SFT ±, 95%1.71.61.61.61.7
Figure 5.Kolibri's mid-training solutions against one hand mix, at 84B tokens on the proxy, one run each. These are printed solves, not a curve. S2 and S3 have nearly the same mix and differ by 1.2 after fine-tuning. German instruction sits at its cap in all of them.

Without the penalty, English web, German web, agentic code and Specialized domains all get 0%, and STEM documents 0.3%. The report’s guess is that the aggregate weights math and code two to one over knowledge. It adds that domains enter the objective through their effect on benchmarks, and Specialized domains “through none of them”, so the objective gains nothing directly from tokens given to that domain.

The other shares in S1 appear to come from the caps (our inference, from domain means). The report says the solve assumed a 30T pre-training and a 5T mid-training, and gives German instruction 1.4% under those caps. Scaling Table 12’s pre-training passes by 1.5 gives English instruction about 30.4% and STEM QA about 24.6%, against 31.0 and 25.1 in S1. A domain mean runs a little low here, since it understates a sum of per-dataset caps. So S1 looks like: fill three domains to their caps, give code nearly all the rest. S5 shows the same pattern at 31.5 and 25.5, and production at 31.4 and 25.4. Under the caps of the 20T pre-training that ran, those domains could have reached 52.9% and 27.9% at 5T, and the run then read only 3.44T. So the two largest shares in the shipped mix may be set by caps computed for budgets the run didn’t use. The report doesn’t say why the solve assumed 30T.

Aleph Alpha then chooses a penalty, without help from the scores: “At 84B, the SFT scores of the solutions with penalties 0, 0.01, and 0.05 lie within 0.9 points of each other, below the resolution of the eval-noise model, so the choice of the penalty rests on the domain weights.” (The printed S1 and S4 cells differ by 1.0; the report works from unrounded scores.) It takes 0.01, “the smallest tested penalty at which every domain keeps a positive weight”. If mid-training’s penalty has the form above, any positive λ\lambda would meet that test, and 0.01 is simply the smallest non-zero value tried.

That budget is one where the report says mixes don’t separate after SFT, so this is a failure to detect. Taking S3, the like-for-like solve, the SFT range is 2.1, close to what the noise model can resolve, and the checkpoint averages have an interior peak (S4 61.1, S3 62.9, S1 61.8), against a yardstick of 0.6 for near-identical runs. The fit’s own optimum, S1, doesn’t win. That’s one path of single runs, and it hints that some penalty helped (our reading).

The final mix also depends on the prior. Aleph Alpha picked C3 from 400B proxy runs, in which C3 led C2 by 3.5 points after SFT. With C3 as prior, the solve takes German web from 18% to 2.1%, and the two web domains end at 3.4% between them. Olmix recommends λ=0.05\lambda = 0.05, saying “noise from regression fitting and proxy-to-target model transfer makes moderate regularization beneficial”. Kolibri uses 0.05 in pre-training and 0.01 in mid-training, and these don’t compare: λ\lambda has the objective’s units, and here those are BPB in one case and a benchmark score in the other. Poolside publishes neither its λ\lambda nor what’s in its prior.

Solved share 0.49; true score 0.077 BPB against the true best 0.076 at 0.58.

Over the last 20 swarms at these settings: 0.49 to 0.61.

Dots: simulated proxy scores. Dashed: the target's true score. Solid: the log-linear laws fitted to the dots, summed. Ticks in the strip: the solved mix in each of the last 20 swarms.

Figure 6.A toy with invented numbers. Few runs make the solved mix wander, and more runs settle it. With proxy bias on, more runs settle it confidently on the wrong share, and a penalty helps only if the prior is closer to the truth.

The ideas are older than these papers, none of which cite the following (our mapping). A score as a function of shares on a simplex is a mixture experiment (Scheffé, 1958), and fitting it is a response surface (Box and Wilson, 1951). An expert merge is one round of federated averaging (McMahan et al., 2017), and its second-order gap is the leading term of what that literature calls client drift (Karimireddy et al., 2020).

Repetition

In Olmix’s example, a mix with 40% code when code is 5% of the data means oversampling it eight times. A solver gives tokens to the sources that score best, and those tend to be small, which is why runs limit passes.

The usual basis is Muennighoff et al. (2023): under fixed compute, up to 4 epochs of repeated data “yields negligible changes to loss compared to having unique data”. They model the value of EE epochs as 1+R∗(1−e−(E−1)/R∗)1 + R^*(1 - e^{-(E-1)/R^*}) fresh epochs, with a fitted R∗=15.39R^* = 15.39. Four epochs are worth about 3.73, 16 about 10.6, and the limit is about 16.4 (computed), a limit the paper warns misses failing runs at 44 epochs, where the real loss is worse. Those are GPT-2-style models up to 8.7B parameters on C4, a web corpus. They say “we always repeat the entire available data rather than subsets of it”, and the downstream results are natural-language tasks. Using four epochs per source within a mix extends that result (our reading). They also expect that repeating earlier does less damage than later, while Kolibri’s cap lets late stages repeat.

Per-source evidence is thin. Feng et al. downsample every source by one fixed factor, so a blend keeps its epoch counts on a small run, and vary epochs directly. Math at 1, 4, 8 and 12 epochs gives averages of 51.49, 51.92, 52.28 and 51.63, with no intervals, on 8B at 1T. They recommend 6 epochs for high-quality crawl and 8 for math and task data.

A source with NiN_i tokens at share xix_i of a budget RR is read xiR/Nix_i R / N_i times, so a limit of mm passes means xi≤mNi/Rx_i \le m N_i / R. In Kolibri’s pre-training m=3m = 3 and R=20R = 20T, so the 80B of Specialized domains can take at most 1.2% (computed). In mid-training, a dataset’s draws across both stages stop at four epochs, so its mid-training cap is four passes minus what pre-training used. In the solve the cap applies to whole domains. A separate step, a waterfill, then works inside each domain: each dataset gets its share, one above its cap is pinned there, and the excess goes to the others in proportion. In symbols, yi=min⁡(κi,c pi)y_i = \min(\kappa_i, c\,p_i) for requested shares pip_i, the same form as the capped tilt above.

Domain's budget 80B of 315B the caps allow: 1 of 5 datasets pinned.

The slider stops at the sum of the caps. Beyond it, the domain receives less than its weight.

Figure 7.A toy with five invented datasets. The rule is Aleph Alpha's: pin at the cap, and redistribute the excess in proportion among the unpinned. Kolibri's mid-training data was built at 5.2T, with seven of 87 datasets pinned, and the run stopped at 3.44T.

Small runs read too few tokens for repeats to appear. At 3B tokens from a 29.3T pool, or at Poolside’s 60B, almost nothing can repeat, so a fitted coefficient says what a source is worth at about zero passes, and the solve applies it at up to three. Aleph Alpha says it directly: “At the 400B and 90B budgets no repetition cap binds.” In pre-training, the size prior and caps account for repeats. In mid-training the prior is a hand mix with no sizes in it, so only the caps do.

Aleph Alpha prepared 5.2T tokens for mid-training, with seven datasets pinned, and stopped at 3.44T, so those datasets got about two thirds of their allowance (3.44/5.2). Across the 3.44T, STEM QA averages about 2.0 passes (25.4% of 3.44T, over 431B) and German instruction 1.6 (1.9%, over 41B). Those are domain means for that stage alone, and they hide pinned datasets and passes from pre-training.

Aleph Alpha’s blog gives an example of concentration inside a domain. German text is about 4.3T seen in pre-training, and “each German token was seen 1.8 times on average”, so about 2.4T unique (computed). The German domains hold about 6.85T, so if those figures align, the run read about a third of what’s there, repeated. (The blog’s 21.3% is the report’s too; the card’s table gives its German domains 23.9%, and no source explains the gap.)

Poolside names repetition as one of two bottlenecks on M.1, with allocation the other, and answers with wider web filters, synthetic rephrasing at scale, and AutoMixer. It publishes no pass count for pre-training. DeepSeek filters out model-generated content “with limited information gain”, such as low-quality machine translation, calling it “implicit duplication” that “may become detrimental over long training horizons”. That points the other way from Poolside’s rephrasing, though the two describe different data (our reading), and the synthetic data article takes this up. Aleph Alpha is the only one of 16 labs that we found publishing a cap.

Gains and losses from fitted mixes

Most of what’s published is one four-way comparison, on Kolibri’s 11.9B proxy at 400B tokens: a run continuing pre-training, hand mixes C2 and C3, and the production mix.

Figure 8.Four runs of 400B tokens on Aleph Alpha's proxy, one run per mix. The report's differences come from unrounded scores, so they can sit 0.1 from what the printed cells give. Against continuing, other benchmarks rise (MMLU-Pro 3.6, PIQA 2.9, MMLU 2.4, Global MMLU 1.7, ARC 1.5) and HellaSwag falls 3.9; the panels show the two largest falls. Intervals are eval noise from replicate fine-tunes. Averages have none.

After SFT, production leads the continuing run by 11.5 points, and the report splits that: 4.3 to C2, 3.5 to C3, 3.8 to production. Only the 3.8 compares a fit with a hand mix. The report’s noise model resolves a gap of 2.6 here and puts this one at p<0.01p < 0.01. Three limits apply. Each mix has one base run, so that noise covers fine-tuning seeds and evaluation only. Half-widths of 1.9 and 1.8 give a standard error of 1.34, and a run-to-run standard deviation of about 1.0 per run would bring 3.8 down to the 5% line (computed). Nothing measures that at 400B. At 84B the near-identical pairs (S2 and S3, S5 and S6, L1 and L2, L3 and L4) differ by 0.4 to 1.2 after SFT, no more than eval noise alone predicts, though the report doesn’t state their seeds. C3 was picked on these same 400B runs. And production’s weights came after a hand edit.

The first step, the 4.3, crosses more than a change of mix: the continuing run stays at 16k and the others go to 64k, and the mid-training pool is decontaminated while the pre-training pool isn’t. The pools differ in how much benchmark text they contain, which the filtering article covers. In production pre-training “22 % to 95 % of the completions are recited” on HumanEval, and English pass@1 falls from 80.0 to 70.0 within 0.3T of mid-training as that recitation drops. The report therefore gives the aggregate without HumanEval too. Over the full model, from the M4 at the end of pre-training to the M4 at the end of mid-training, the aggregate rises 2.8, or 2.9 without HumanEval, and these cross the length change as well. The report prints no full-scale run of this model on another mix to compare with.

The losses need the same split. TriviaQA ends 8.4 points below continuing on the production mix (the report’s figure), and the hand mixes end 6.2 and about 8.4 below it, so against its own prior the fit costs about 0.1. These are gaps to a run that kept reading other data, and say nothing about forgetting by themselves. Forgetting needs a before and after, which the full model gives: M4 to M4, TriviaQA falls 6.8, German HellaSwag 5.6 and HellaSwag 2.6, while MMLU-Pro rises 2.9 and ARC 1.2. The two knowledge means fall only 1.0 in English and 1.4 in German, with single benchmarks inside them falling further. The report says “the cause of the TriviaQA loss remains open”, pointing to the difference in sequence length. Memorised text could play a part too (our reading), though at natural dilution, on the proxy, the report found contamination “not measurable”.

German HellaSwag is the loss that does separate the fit from its prior. “At 400B, the HellaSwag (DE) score of the production mix lies 5.5 points below that of C3, a cost of 0.09 points in the aggregate score.” It rises with German web across the 84B runs, from 65.7 at 0% to 79.4 at 19%, though German web also moves against code in those sweeps, and which German data is used matters too. The 0.09 is the benchmark’s weight: 0.2 for German, times 1/3 for knowledge, times 1/4 for four German knowledge benchmarks, is 1/60, and 5.5/60 is 0.09 (our derivation). A gain of 0.69 on one English math or code benchmark, with weight 1/7.5, pays for it. Hence the report’s conclusion: “Preserving a capability thus requires its own term or constraint in the objective.” The benchmark was in the objective all along, at a low weight. A domain with no path into the objective is the other case, and Specialized domains show it.

Figure 9.Percent change against the manually designed prior mix, on "a 3B parameter model for 1.5T tokens". These are relative changes with no absolute scores, intervals or seed counts, so a large percentage can be a small move from a low score. This test read 1.5T, against more than 30T for XS.2's run, and the report doesn't say whether its 3B model has XS.2's shape. The held-out benchmarks are code and math, like the main targets.

Poolside reports relative changes on a small test against its manual prior. Commonsense falls on all four benchmarks, which it calls “not surprising given the optimization objectives we chose”, and its optimisation benchmarks are “focused on coding”, though they include GSM8K and MMLU. XS.2’s mix has 30.6% raw code and 25.2% web, and no full-scale effect of the mix is published. ARC moves opposite ways at the two labs, which is a reason not to speak of one “commonsense” loss. Olmix likewise reports gains in BPB: its abstract’s “11.6%”, for reusing a mix as datasets change, isn’t accuracy.

Small budgets can hide differences. Kolibri’s mixes “separate after SFT at the larger proxy budget only”. At 84B, 14 of 120 pairs among 16 fine-tuned runs differ beyond noise, against 6 expected at the 5% level. The report estimates two thirds of those scores’ variance as noise. With 16 runs that share could lie anywhere from about a quarter to all of it, and it counts fine-tuning seeds only (computed). The pairs also share runs, so 14 is less surprising than it looks: with no real differences, about one draw in ten reaches 14 (computed by simulation). At 400B every adjacent pair differs. On those four runs the checkpoint average “appears to understate” what fine-tuning shows, though it’s a different benchmark set.

Objectives differ too, and no source compares them. Aleph Alpha fits BPB in pre-training and benchmark scores in mid-training. Poolside uses benchmark scores. Olmix uses BPB on task text, and MergeMix uses scores because lower perplexity does “not reliably translate”. Aleph Alpha’s upper-bound test shows that both can move with memorised text: adding removed documents back, 8.2% of the tokens, “inflates HumanEval by 42 points, MMLU by 29 and TriviaQA by 28 at the same step, halves the bits per byte on the benchmark text”.

Moving between mixes

A run can switch mixes at once or blend them. Aleph Alpha blends, taking the old mix’s share down linearly from 1 to 0 over the first H=100H = 100B of the 3.44T tokens, so the old mix supplies 50B. Those tokens come from beyond what the 20T run read, so they’re new. The report cites prior work: “retaining a share of the previous data limits the forgetting” that a shift causes.

In the report’s one test, on the 84B proxy, mix C1 through the transition has a checkpoint average of 61.1, and switched directly 61.6. Without HumanEval they are 59.7 and 61.0. The test can’t settle this question. That score moves mostly with math and code, with the knowledge mean spanning 1.2 points against 5.8 for math, so it’s a poor measure of forgetting, and any window means reading less of the new mix before scoring. The report doesn’t say what window the 84B run used, the transition run had no fine-tune, and C1 sits closer to the pre-training mix than production does. The report draws no conclusion and keeps the transition.

Rehearsal, keeping some old data in the later stage, appears at other labs. Mistral tunes it jointly with the learning rate and settles on “1/5 of the peak pretraining LR and 1/10 of pretraining rehearsal”, and Nemotron 3.5 Super has 20% replay in its mid-training blend. The mid-training article covers both.

Packing

At a sequence length of 16k, a 200,000-token document fills 13 sequences. Packing turns a mix’s tokens into fixed-length sequences, and a poor method can leave a few optimiser steps dominated by that one document, even when the mix is broad. The simple method concatenates documents and splits at every TT tokens, with two problems that Aleph Alpha’s report names. “Almost every sequence boundary falls inside a document”, so short documents get cut for no reason. And consecutive pieces of a long document land in adjacent microbatches (the parts of a batch on one GPU), so “for a few optimiser steps the gradient is dominated by one document, one domain and one writing style”. Shuffling documents first doesn’t help, since each document’s pieces still come out back to back.

concatenatebuffer+ packer
Needless cuts16240
Longest run of rows dominated by one document622
Padding0.0%0.0%5.9%

Each row is one sequence, each colour one document, and strong coral is the long document. A tick at the end of a piece marks a cut. A bar at the left marks a row where one document holds more than half. Hatched cells are padding: this toy pads where Kolibri's packer overfills a sequence and drops the excess.

Figure 10.A toy with invented document lengths. Reports measure padding and loss smoothness for these methods, and we found no lab in the map publishing a benchmark change from packing.

Best-fit packing (Ding et al., 2024) treats the first problem as bin packing, offline: put each piece in the fullest sequence where it fits, longest first. A cut then happens only for a document longer than TT, and sequences hardly increase (under 0.01%). Their packed arm also masks attention across documents, so tokens can’t attend to an unrelated neighbour, and the concatenated baseline doesn’t. On a 13B model at 2k, reading comprehension rises from 46.71% to 48.92% in the combined comparison, and from 47.92% against a baseline that also has the mask, so about 1.0 point belongs to packing itself.

For the second problem there’s the Random Sequential Document Buffer (RSDB) of Arcee’s Trinity report, a model outside this site’s map. It keeps thousands of open documents and reads the next chunk of a random one, so the pieces of one document spread across many steps. A small internal test shows batch heterogeneity (largest microbatch loss minus the mean) down 46%, with no benchmark.

Aleph Alpha combines them online for Kolibri’s base stages: an RSDB of 4,096 documents per worker, then a packer that fills each sequence from 1,024 pending chunks and drops tokens past the end when no subset fits, at a rate it doesn’t print. It reports “a smoother training loss”, with no number.

SFT keeps conversations whole, which leaves padding. Next-k-Fit (Johnson, 1974) holds kk open bins, and more bins take more memory and start-up time. At sequence length 32k, raising kk from 1 to 1,024 lowers padding from 14.8% to 0.6%, and Kolibri’s run at k=256k = 256 and 256k has 0.001%. NVIDIA does length-aware best-fit for SFT. DeepSeek pre-splits very long documents in pre-training and reaches “a padding rate of at most 10⁻⁴” on V4.1-Flash. Labs split in one stage and keep whole in another because the data differ (our reading; no source says so).

Eight of 16 labs mention packing in training: Aleph Alpha, DeepSeek, NVIDIA, Poolside and Liquid for fine-tuning or base stages, Mistral and Xiaomi for RL, and Z.ai for building synthetic long-context data. Three name a fit rule (Aleph Alpha, DeepSeek, NVIDIA). None publishes a benchmark change. Poolside reports a cost in a mixture-of-experts model: on M.1, about 5% of pre-training tokens were padding, and “all padding tokens in a batch are routed to the same expert at the same time, potentially saturating that expert”.

Every budget in this article is in tokens, and a tokenizer sets how much text a token holds. On one German web corpus, Aleph Alpha’s figure of twelve tokenizers runs from 3.28 bytes per token (Kimi K3’s) to 4.90 (Kolibri’s), and on English from 4.16 to 4.67. So 1T tokens isn’t the same amount of text at two labs. Vocabulary size doesn’t settle it: Gemma’s 262,144 entries get 4.13 on the same German, and Kolibri’s 128,000 get 4.90.

Against GPT-5’s, the best of the nine other tokenizers on German at 4.35, Kolibri’s needs 11.2% fewer tokens there, and GPT-5’s wins on English, 4.67 against 4.58. The method, UniBPE, keeps the greedy merging of byte pair encoding (BPE) and scores merges by a different loss. Plain BPE trained on the same corpus with the same vocabulary reaches 4.89 and 4.59, so the compression seems to come mostly from training on that corpus (our inference). The report credits UniBPE with morpheme boundaries and better long-context recall. Its card prints 4.7 and 4.2, with no explanation of the gap.

Three of 16 labs state that a tokenizer was trained on their own data (Aleph Alpha, Liquid’s LFM2, Poolside), and four reuse an in-house one (Google, OpenAI, DeepSeek, Qwen’s Omni model).

The methods compared

MethodAnswersDecided fromCostsBest evidence, and its scope
Stage shiftGood data is scarceMostly unstatedEarlier skills can fade; length often changes tooFeng: +1.74 points over random order, 8B at 1T
Proxy swarmToo many sources to set by handSmall fresh modelsGives up what the objective weights lightly; noisy fit; scale gapOlmix: 12.2% better average BPB than natural, 1B at 100B. Laguna: Table 3, relative, 3B at 1.5T
Expert mergesRuns from a base are costlyWeights from a shared baseMay not rank; only first-order, at equal lengthWang, mid-training: 0.92 over 12 mixes. Kolibri, SFT: 0.77 to 0.94 with 2 or 3 specialists, −0.09 over 6 with 20. None printed for Kolibri mid-training
KL penaltySolver empties sourcesPeople: a prior and a λ\lambdaResult rests on bothKolibri: 4 of 9 domains at 0 without it; SFT unresolved at 84B
Fit against hand priorDoes fitting beat judgement?400B proxy runs−5.5 German HellaSwag, on one run per mixKolibri: +3.8 after SFT, against the prior it was built from
Repetition capSolver over-weights small sourcesA count of passesBasis is whole-corpus, at GPT-2 scaleMuennighoff: 4 epochs, 8.7B on C4. Feng: per source, 8B at 1T
TransitionAbrupt shift may cause forgettingPrior workTokens not on the new mixKolibri: 61.1 against 61.6 at 84B, not a test of forgetting
PackingCuts, and runs of one documentPadding, loss smoothnessBuffer memory, start-up time, dropped tokensDing: reading +1.0 point net of the mask, 13B

Where the models in the map sit

Token counts below aren’t like for like: some count seen tokens and others a corpus, with or without images, and some totals include the later stage. Each row names the model whose report states it.

LabLater base stage on different dataStated content of the later stageHow the mix was set
Aleph Alpha (Kolibri)20T, then 3.44T, then 200BMath, code, reasoning, instruction. Then long documents and OCR’d PDFsFitted. Pre-training: KL to a flattened size prior. Mid-training: KL to a hand mix, then a hand edit
Poolside (Laguna XS.2)Partial: more than 30T, “stage-1 mix” undefined. It files 200B long-context and about 60B “mid-training” under post-trainingMid-training 40/30/30 reasoning, code, chatPre-training fitted on XS.2, manual on M.1. Mid-training tuned, no method
NVIDIA (Nemotron 3 Ultra)15T + 5T inside pre-trainingQuality tiersUnstated
Mistral (Large 4)65% + 35% inside pre-training, by design, unit unstatedMore code and scienceUnstated
Liquid (LFM3)22T including 3.5T lastCurated, agentic tracesUnstated
Xiaomi (MiMo-V2-Flash)22T + 4T + 1TCode up, about 5% synthetic reasoningUnstated
Z.ai (GLM-5)27T + 1T + 500B + 50BLong documents, agent trajectoriesUnstated
Moonshot (K3)Partial: long documents and videos upsampled in cooldown“Ablation studies on smaller models”, no method
DeepSeekPartial: agentic data in V4’s mid-training“Optimize the data mixture” (V4.1), no method
Qwen (3.5-Omni)Partial: more long audio and video in its last pre-training phaseUnstated
Google, OpenAI, Thinking Machines, Tencent, MiniMax, StepFun

A blank means searched and not found. Poolside’s “mid-training” is at most 0.2% of its pre-training tokens and Aleph Alpha’s is 17% (computed), so the label doesn’t say how much of a run it covers.

Easy to get wrong

  • “Optimised” doesn’t mean tested at scale. Only Kolibri prints a full-scale chain of scores, for the mix that ran. Neither lab reports a same-model full-scale run on a second mix. Each has an earlier model on another mix (Kolibri Origin, Laguna M.1), at a different size and budget.
  • Most of Kolibri’s 11.5-point lead belongs to leaving the pre-training recipe and to hand tuning. The fit’s part is 3.8.
  • A domain at 0% may only have no benchmark that rewards it, as with Specialized domains, so the solver’s weights reflect the objective, more than the data.
  • The >0.9 Spearman for merges is Wang’s, over 12 mixes in four domains. Aleph Alpha’s own figure at 20 clusters is −0.09, over six, and it prints none for mid-training.
  • Muennighoff’s four epochs come from repeating whole corpora on natural-language tasks. Kolibri uses 3 per source in one stage and 4 per dataset across two.
  • Poolside’s +43% and Kolibri’s +3.8 points differ in unit, baseline, model and budget, so they can’t be compared.

Check yourself

  1. A source holds 500B tokens. With Kolibri’s pre-training limit of 3 passes at R=20R = 20T, what’s the largest share it can take? How many passes would a 20% share mean?

    Answerx≤mN/R=3×0.5/20=7.5%x \le mN/R = 3 \times 0.5 / 20 = 7.5\%. A 20% share is 4T tokens, which is 8 passes over 500B.
  2. Using xi∝qie−si/λx_i \propto q_i e^{-s_i/\lambda}, why can’t a domain get exactly 0% at λ=0.01\lambda = 0.01, assuming mid-training’s penalty has this form? And why can the solve give 0% with no penalty?

    AnswerThe exponential is positive at any finite si/λs_i / \lambda, so a domain with qi>0q_i > 0 keeps some share, even one small enough to round to nothing. With no penalty, nothing pushes a share off the boundary, and the fit alone can leave a domain at zero when its slope is worse than the slopes of the domains in use.
  3. German HellaSwag has weight 1/60 in Kolibri’s aggregate, and an English math benchmark has weight 1/7.5. What does a 5.5-point drop on the first cost in the aggregate, and how many points on the second offset it?

    Answer5.5/60≈0.095.5/60 \approx 0.09, and 0.0917×7.5≈0.690.0917 \times 7.5 \approx 0.69 points. A mix with that gain scores the same on the sum, so a skill you want to keep needs its own term or a constraint.
  4. Why does the argument for merging need a shared start and a short horizon, and what doesn’t it cover?

    AnswerWith a shared start, gradients are taken at one point and a mix of them equals the gradient of the mixed loss. The approximation is first order, so the second-order gap, of size (ηn)2(\eta n)^2, must stay small. It compares runs of equal length, so it doesn’t cover a merge standing in for a longer run: that the ordering survives is an empirical claim.
  5. Why is a Spearman of −0.09 over six weightings not evidence that merges never rank?

    AnswerSix ranks give an interval of about −0.84 to 0.78, each trained score is one run, and the sweep changed two things at once (number of specialists, length of training). It rules out agreement near Wang’s 0.92 in that setting, and that’s all.

In the map

Methods

Taxonomy nodes

Further reading

↑