Kolibri, Aleph Alpha’s 78B-parameter model, reads 20T tokens in its main stage, drawn from 78 datasets that hold about 29.3T tokens between them. Someone has to decide how many of those 20T come from each dataset. That’s 77 parameters to the data mix (the last share is whatever remains), and you only want to train once, rather than doing multiple full scale runs and keeping the best.
Some labs change the mix between stages, keeping broad web text for early training and spending scarcer, better data late. A few also fit the proportions, by training a swarm of small models on different mixes and solving for the mix a fitted law says is best. Two of the 16 labs on this site, Aleph Alpha and Poolside, publish a method for fitting.
A solver with no penalty gives four of Kolibri’s nine mid-training domains exactly no tokens. To prevent that, Aleph Alpha adds a penalty toward a mix people chose in advance. The fit then gives up what its objective weights lightly, and part of what comes out is that prior.
Background
A mix gives each data source a share of a stage’s tokens. Sources group into domains such as web text, code or math. A run’s budget is the number of tokens it reads, counting repeats, and those are tokens seen. A source’s pool is what it holds, and unique tokens are the distinct ones actually read. A share of a budget divided by the pool gives passes: 4T seen from a 500B source is 8 passes, and epochs mean the same thing.
Runs come in stages. Pre-training is the long first stage on broad text. Mid-training is a shorter base-model stage on better data, and long-context extension lengthens the sequences. Supervised fine-tuning (SFT) then trains on conversations and turns a base model into an assistant. Scores taken after a short test SFT are closer to what ships, so reports use them to judge a base mix. Each lab draws these lines its own way.
As in the primer, a proxy is a small model standing in for a run nobody can repeat, and an ablation changes one piece of a recipe to see what it did. A checkpoint average combines the weights from a run’s last few saves, to smooth out step-to-step noise. M4 means four saves. A wider window scores higher by itself, so only same-size averages compare. The reports call these merges, and we keep that word for something else below.
Bits per byte (BPB) is cross-entropy loss per byte of benchmark text, where lower is better. Accuracy benchmarks count right answers: GSM8K is math word problems, HumanEval is small coding tasks, MMLU and TriviaQA test knowledge, and HellaSwag and ARC test commonsense reasoning. Kolibri’s aggregate score averages knowledge, math and code in each language, then weights English 80 and German 20. The report’s knowledge group includes HellaSwag and ARC. Its SFT score is a different number: the mean over 11 benchmarks after one fixed test SFT. Repeating that test SFT from the same base with a new seed gives a different score, and that spread is the eval noise the report reads differences against.
Modeling code records none of this, so we read technical reports first, then cards and official blogs. Kolibri’s report is the most detailed public account and our worked example, and its choices may not be typical. Reports describe what shipped, and only two of 16, Poolside and Aleph Alpha, tell of a mix or a mixing method that fell short.
Later stages get different data
Many labs change data between stages with no optimiser at all. Of 16 labs, six state that a later base-model stage uses different data (NVIDIA, Mistral, Aleph Alpha, Liquid, Xiaomi, Z.ai). Four more say something partial (Poolside, Moonshot, DeepSeek, Qwen). We found nothing on it from the other six, which doesn’t mean they don’t do it.
High-quality data is small, so a lab can’t put a whole run’s budget on it without repeating it many times. Hu et al. (MiniCPM, 2024) mixed instruction-style and high-quality data into only the last stretch, the learning-rate decay of about 10%, and scored models after an identical SFT. On a 2.4B model, MMLU rose from 44.6 to 50.9 and GSM8K from 27.7 to 42.3. On a 1.2B model most scores rose too, but GSM8K fell from 34.2 to 31.8. No seeds or intervals are given. A longer SFT alone (12B against 6B tokens) moved almost nothing, so they rule out a shortage of SFT tokens and recommend that specialisation “should start from the decay phase”.
Feng et al. (2024) made this a recipe with a controlled comparison. On an 8B model at 1T tokens, the same source weights in two phases beat the same weights in random order: an average score of 52.86 against 51.12, which is 1.74 points, or the paper’s relative “3.4%”. The gain is mostly GSM8K (+9.47) and code (+2.78), and MMLU slightly drops. Among the lengths tried, 40% in the second phase scores best. They report no error bars, and their 15T run has no baseline at 15T.
Scarcity explains why good data gets a small share, and doesn’t explain why it goes last, since Feng’s comparison changes only the order. One picture helps (ours, from standard SGD on a quadratic, and networks aren’t quadratic). In a direction of curvature , each later step shrinks an earlier step’s contribution by a factor , with the learning rate. Late tokens therefore count for more in the final weights, and early ones fade, which is the benefit and the cost in one effect.
Labs split at very different points. Nemotron 3 Ultra goes about 75/25 inside pre-training (15T then 5T), and Nemotron 3 Nano switches at 94%. Mistral Large 4 has 65/35 by design, in a report whose Figure 8 caption describes a run at 8×10²³ FLOPs that restarted from the end of phase 1 and “trained for approximately two-thirds of the original phase 2”. Kolibri’s separate stage comes to about 85/15 (3.44T after 19.99T, computed). NVIDIA cites Feng, Aleph Alpha cites Hu, and Mistral says “two-phase curriculum” and cites neither, so we can’t trace adoption from one lab to the next. We found no lab reporting a test of its split point at full scale.
lower tiers: crawl-medium 4.3 to absent, syn-crawl-medium 10.3 to 3.8, finepdfs-unfiltered 4.9 to absent. higher tiers: finepdfs-high absent to 10.7, finepdfs-medium absent to 2.5. general instruction: sft-general. legal: legal, absent then 0.1. Blue segments rise, pale blue by under 2 points. Each panel has its own bars from 0 to 100%; NVIDIA's shares sum to 99.9 as printed.
Labs use the word for different changes. Mistral moves whole domains, code from 16.9% to 39.0% and web from 56.3% to 32.0%. NVIDIA keeps code at 14.0% and swaps tiers, dropping medium crawl and adding high-quality PDFs. Bengio et al. (2009) defined a curriculum as a sequence of reweightings whose weights only grow until the training distribution is reached. Tiers dropping to zero, as at NVIDIA, don’t fit that (our reading). Mistral’s shift can fit it if phase 2 is taken as the target, since a share can fall while its weight doesn’t. So “curriculum” here is loose usage for a mix that changes over time.
Better data late doesn’t hold at every step. Kolibri’s long-context stage swings back toward documents, with 15.8% English web and 33.3% OCR’d PDFs (its card). The stage takes a third of its tokens from a pool of naturally long documents and reweights part of the rest toward length, and “long documents are concentrated in web text and scanned material”. The report calls the resulting shift between domains “a consequence of the length reweighting rather than an explicit design choice”.
No source cleanly measures what a shift costs, and sequence length often changes with the mix. Kolibri goes from 16k to 64k at its switch, Mistral from 4K to 40K, and Liquid and Z.ai extend context in the same stage. Their results can’t separate the two.
Fitting a mix on a swarm
Poolside writes that “optimizing data mixtures manually becomes intractable as the number of datasets, capability trade-offs, and training constraints grows”. On M.1, the model before XS.2, a manual mix gave “excessive repetition in high-value subsets” and “suboptimal allocation of data budget across different sources”.
The published methods follow one schema, named by Olmix (Chen et al., 2026). Train a swarm of proxies on different mixes. Fit a regression from mix to score. Solve for the mix the regression says is best. Mixes are drawn from a Dirichlet distribution, a distribution over shares that sum to one, with a concentration setting how far they stray from a centre. RegMix (Liu et al., 2024) showed that a regression from 1M models can rank mixes on 1B models: rank correlation 0.97 over 64 mixes, on validation loss on one web source, using gradient-boosted trees.
Aleph Alpha’s pre-training search trains 3,354 proxies of 30M parameters, dense with no experts, on 3B tokens each, centred on natural shares (proportional to source size). For each of 24 benchmarks it fits a law of the mix ,
where gives a floor and holds one coefficient per source. Subtract the floor and take logs, and the result is linear in , hence log-linear. The form comes from Ye et al. (2024), who call their laws “empirical findings”, and it reaches Kolibri through Olmix. The report prints no goodness of fit. It then solves
with equal weights , a prior , a Kullback-Leibler (KL) penalty that grows as the mix leaves , and caps that limit repetition. The floor is constant and drops out. Each term is convex and the constraints are linear, so a standard solver finds the optimum, and the hard part is the noise in what it’s optimising. (, , , follow the report and are local to this section.)
The estimate has separate sources of error. The gap between proxy and target is a bias, as is any mismatch between the true response and the law’s form, and more runs fix neither. Noise in the coefficients is variance, which more runs do reduce. Solving over a noisy fit adds an optimistic bias, since the solver picks whichever coefficients happen to look best, so the fit’s own predicted score flatters its choice.
Aleph Alpha’s swarm comes to 10T tokens (computed), which reads like half a run. By tokens times parameters it’s about 0.02% of one (crude, across dense and MoE). The limit is many small jobs and many evaluations. The gap to the target is larger. These models are 2,600 times smaller by total parameters and dense, while Kolibri is a mixture-of-experts (MoE) hybrid, so they can’t show anything a mix does through expert routing. Olmix tested this proxy size against 1B targets (rank correlation 0.896), which stops far short of 78B. Olmix also finds 1M proxies unreliable () and recommends at least 15M, against RegMix’s 1M, while noting that RegMix’s public code is closer to 15M.
The report warns about its own fit: the optimiser “is sensitive to the number of swarm samples, and the prior and its two knobs (λ, T) influence the predicted optimum dramatically”. It also writes that “smaller sources are typically less well-explored even at the lowest concentrations”, and a Dirichlet draw’s variance in a share, for a centre share and concentration , does get smaller in absolute terms as the share does. It had to patch token-count errors, and “the sampling prior for the swarm ablations, computed from the old counts, cannot be corrected after the fact”. The fit also doesn’t follow the filter: “some of our German quality-classifier buckets are down-weighted relative to their labels”.
The report prints no benchmark gain for this mix against any other. It gives a predicted BPB, 3.670 ± 0.002 against 3.667 ± 0.002 for the unconstrained optimum, from the same fit that chose the mix. The card says predictions were “confirmed using a separate validation proxy setup”, with no numbers. At domain level the mix ends near the pool’s shares: English web 43.4% against 41.8%, code 13.6% against 14.8%. Specialized domains (legal and public-sector text) have four times their pool share, close to their cap, and STEM QA two thirds of its. That needn’t mean the fit did little, since its prior isn’t the pool, and neither the prior nor the per-source weights are printed at that level.
Poolside’s AutoMixer has the same shape on another scale: about 60 proxies per ablation, 0.5B parameters MoE on 60B tokens, over more than 50 dataset groups, one non-linear surrogate per capability group, in a form it doesn’t state. Olmix’s minimum for a log-linear law would be over 150 runs there, but that rule needn’t transfer, since Poolside’s search stays inside an L1 ball around its prior, both when sampling and when solving. That keeps the search local, and it avoids fitting in one place and solving in another.
Merging experts in place of runs
A fresh tiny model may not test a mid-training mix. Writing of RegMix’s 1M proxies, Wang et al. (2026) say “such tiny proxies are ineffective for capturing the emergent capabilities” that stage needs. A proper test starts from a trained base, and doing that for every candidate is expensive. MergeMix estimates each candidate with no training on the mix.
From one base , train one expert per domain on that domain alone. These are whole models, unrelated to the experts inside an MoE layer. For a candidate mix , evaluate an expert merge,
Here is why that can work. Write for domain ‘s gradient at . After steps at learning rate , an expert has moved about , so the merge has moved . The gradient of a mixed loss is the mix of the gradients, so a run on mixed data for steps moves the same way, to first order in . At second order, with Hessians , Wang’s terms combine into one line (our rearrangement):
The covariance is taken across domains with weights . It vanishes if the domains share a Hessian or share a gradient, so what matters is how curvature and gradient vary together. Approximation needs times the curvature to be small, along with the shared start and constant learning rate that the paper lists. A shorter horizon reduces this error and leaves less signal over minibatch noise, and merging also averages the experts’ noise, which a single run doesn’t do.
This is a motivation, and not a bound. It also compares a merge with a run of the same length, and neither published use has that: Aleph Alpha’s 20B experts stand in for 84B runs, and Wang’s 5B experts for 200B runs. That the ordering of mixes survives the longer horizon is an empirical claim.
Merged point to the mixed run of the same length: 0.037.
Merged point to the mixed run 4.2× as long: 1.702.
ηn = 0.30: 600 steps at η = 0.0005.
Coral and teal: the expert paths and contours of A and B. Dotted: between the experts' ends, with the merged point (diamond) at share x. The third solid path and its filled dot: the mixed run. Dashed, open circle: the longer mixed run. Thin lines from the diamond are the two gaps.
Wang’s evidence: over 12 mixes in four domains on an 8B MoE model, merges rank mixes as real 200B mid-training runs do, with a Spearman correlation of 0.92 on the overall score (95% interval about 0.67 to 0.98, computed). By capability it runs from 0.72 to 0.98, with 12 mixes too few to resolve those differences. Their chosen mix beats a production manual mix by 0.5 points on the average, with no interval. Their 100× saving counts training tokens against ten manual runs, and leaves out evaluating 40 merges.
Aleph Alpha uses nine experts of 20B each, 180B in all, to stand in for 84B runs on 113 weightings. Those would cost 9,492B as real runs, a saving of 98% before validation (computed). It fits one log-linear law per benchmark to the merge scores. Then 18 real 84B runs and four 400B runs follow, which is how these mixes were actually judged. The report’s “above 0.9” is Wang’s number, and it prints no correlation of its own for mid-training merges.
Its one comparison of merges and runs is on fine-tuning, which Wang didn’t study. An SFT mix is a mix too, over clusters of conversation data. With two or three specialists (experts for one cluster) of 20B, merges and runs agree on two benchmarks, at Spearman 0.77 to 0.94 over four or six weightings. On MATH-500, all scores sit between 84 and 91 and “neither ordering carries information”. Aleph Alpha then applied the method to the whole SFT mix: 20 clusters at 4.5B each, 90B of training instead of 63 runs totalling 1,057B, “91 % less training compute” (about 79% if the eight trained candidates also ran 16.78B each, computed). “With 20 clusters, the merges find good candidates but do not rank them.”
A baseline. B random draw (concentration 50). C refinement. D refinement. E random draw (concentration 15). F refinement. Circles are random draws, diamonds refinements, drawn near the best merges, and the square the baseline. trees: tree proposal, no merge score. GP: Gaussian-process proposal, no merge score.
Six ranks are few: at a correlation needs to reach 0.89 to clear the usual 5% test (computed). So −0.09 tells us this ranking wasn’t near Wang’s 0.92, and can’t separate zero from moderate agreement. The search also moved toward high merge scores, and refinements centred there could select for the proxy’s errors (our reading). The report says “our experiments do not show which of these two differences breaks the ranking”, and the differences are 20 specialists instead of two or three, and 4.5B of training instead of 20B. Aleph Alpha therefore trained eight weightings and chose on trained scores. Six of the seven besides the baseline train above it, up to 56.7 against 51.5, in one run each. That’s weak on its own: if all eight were equal, at least six would beat the baseline one time in four (computed). The best was a random draw from mid-ranking, so we can’t credit the merge over the sampling.
Here are the searches side by side, for base-model mixes.
| Kolibri pre-training | Kolibri mid-training | Laguna XS.2 AutoMixer | |
|---|---|---|---|
| Proxy | 30M dense, 3B tokens | 11.9B total, 0.65B active MoE, from a 2.79T checkpoint | about 0.5B MoE (total or active unstated), about 60B tokens |
| Runs | 3,354 | 9 experts of 20B, 113 merges, then 18 runs at 84B and 4 at 400B | about 60 per ablation; number of ablations unstated |
| Dimensions | 78 sources | 9 domains; datasets inside a domain by a fixed rule | more than 50 dataset groups |
| Metric | mean BPB, 24 benchmarks (18 English, 6 German) | aggregate benchmark score | benchmark evals in 5 capability groups |
| Law | log-linear, per benchmark | log-linear, per benchmark | non-linear surrogate per group, form unstated |
| Penalty | KL to a flattened size prior, | KL to a hand mix, | KL to a manual prior, plus an L1 ball; values unpublished |
| Repetition | at most 3 passes at 20T, per source | at most 4 passes across both stages, per dataset | no number published |
Two other labs mention proxies for mixes with no method. Moonshot sets “domain-specific sampling rates determined by ablation studies on smaller models”, and DeepSeek’s V4.1 report designs “a scaling ladder over model parameters and training data”, without saying that it sets any weight. Mistral uses that phrase for its recipe and not its mix. NVIDIA tests one dimension at a time, such as an Indic share at 10, 20 and 30%, and says these runs “inform mixture design but do not establish an optimal share for Super”. Olmix and MergeMix are external work that Aleph Alpha adopts, and Poolside cites Olmix and RegMix among its inspirations.
How the penalty and the prior set the mix
The penalty is , which is zero at . In Kolibri’s pre-training solve the prior is (the report’s temperature , unrelated to token budgets). Larger sources get more, though much less than in proportion: a source 1,000 times larger gets 10 times the prior share. Note that the swarm samples around natural shares, while the solve pulls toward this flatter prior. “The KL pull trades quality against repetition: at λ = 0 the solver ignores source sizes and picks only the best-fitting sources, while a large λ reproduces the size prior.”
The penalty’s effect has an exact form (our derivation, not in the report). First take the objective’s slope as fixed at for source . A multiplier for gives , so
The penalty tilts the prior exponentially toward sources with a good slope. On the real objective the slopes depend on the mix, and caps apply, so the solution satisfies the same form as a fixed point:
with set by . Three things follow. For any no share is exactly zero, though it can be tiny: with , a slope gap of 0.2 gives a factor near . At , a single benchmark with no caps would put everything in one source, since a linear function on shares is lowest at a corner. With caps it fills the best sources in order. With several benchmarks, slopes equalise across the sources in use below their caps, so several can survive.
The third follows from counting. The first term sees the 78 shares only through 24 numbers, . So at least 53 directions in the mix leave the fit unchanged (computed), and along them only the prior and the caps decide. Among the mixes that score the same, the solver returns the one closest to the prior in KL that fits under the caps. This form is old: statisticians call it exponential tilting, or the Esscher transform (Esscher, 1932), and one step of it is exponentiated gradient (Kivinen and Warmuth, 1997).
Mid-training gives a printed series. Its mixes need names, since the report’s labels appear in the tables.
| Name | What it is | Prior, |
|---|---|---|
| C1, C2 | Hand mixes with identical domain shares, differing in how datasets are sampled inside a domain | |
| C3 | Hand mix with more instruction, STEM and agentic code, chosen from 400B proxy runs | |
| S1, S2, S4 | Solves against C2 | 0, 0.01, 0.05 |
| S3 | Near-twin of S2, projected from a 5T budget, as S1 and S4 are | C2, 0.01 |
| S5 | Solve against C3 | C3, 0.01 |
| Production | S5 with German instruction raised by hand to its cap, then trained on |
The report prints neither the law nor the penalty for mid-training, and that solve maximises a score, so the formulas above carry over only if the form does. A mid-training mix also has a second axis the solve leaves alone: datasets within a domain follow a fixed rule, since optimising them “requires one expert per dataset, 87 trainings in total”.
λ = 0.01 (S3, projected). These steps are all that's printed.
At or near the computed 5T cap: English instruction, German instruction. Outline: hand mix C2. Triangle: the cap at 5T under a 30T pre-training, computed, for the domains where it may set the share (our inference); filled when at or near it.
All five solutions, as a table
| % | C2 | S4 | S3 | S2 | S1 |
|---|---|---|---|---|---|
| English web | 33.0 | 20.3 | 3.1 | 3.1 | 0.0 |
| German web | 19.0 | 2.3 | 0.4 | 0.4 | 0.0 |
| English instruction | 10.0 | 14.9 | 30.7 | 31.5 | 31.0 |
| German instruction | 1.0 | 1.4 | 1.4 | 1.4 | 1.4 |
| Code | 10.0 | 27.1 | 29.8 | 29.4 | 42.1 |
| Agentic code | 8.5 | 9.7 | 7.9 | 7.8 | 0.0 |
| STEM documents | 6.5 | 13.7 | 10.9 | 10.8 | 0.3 |
| STEM QA | 9.0 | 9.6 | 15.7 | 15.5 | 25.1 |
| Specialized | 3.0 | 1.0 | 0.1 | 0.1 | 0.0 |
| M10 | 60.9 | 61.1 | 62.9 | 62.4 | 61.8 |
| SFT | 40.6 | 39.9 | 42.0 | 40.8 | 40.9 |
| SFT ±, 95% | 1.7 | 1.6 | 1.6 | 1.6 | 1.7 |
Without the penalty, English web, German web, agentic code and Specialized domains all get 0%, and STEM documents 0.3%. The report’s guess is that the aggregate weights math and code two to one over knowledge. It adds that domains enter the objective through their effect on benchmarks, and Specialized domains “through none of them”, so the objective gains nothing directly from tokens given to that domain.
The other shares in S1 appear to come from the caps (our inference, from domain means). The report says the solve assumed a 30T pre-training and a 5T mid-training, and gives German instruction 1.4% under those caps. Scaling Table 12’s pre-training passes by 1.5 gives English instruction about 30.4% and STEM QA about 24.6%, against 31.0 and 25.1 in S1. A domain mean runs a little low here, since it understates a sum of per-dataset caps. So S1 looks like: fill three domains to their caps, give code nearly all the rest. S5 shows the same pattern at 31.5 and 25.5, and production at 31.4 and 25.4. Under the caps of the 20T pre-training that ran, those domains could have reached 52.9% and 27.9% at 5T, and the run then read only 3.44T. So the two largest shares in the shipped mix may be set by caps computed for budgets the run didn’t use. The report doesn’t say why the solve assumed 30T.
Aleph Alpha then chooses a penalty, without help from the scores: “At 84B, the SFT scores of the solutions with penalties 0, 0.01, and 0.05 lie within 0.9 points of each other, below the resolution of the eval-noise model, so the choice of the penalty rests on the domain weights.” (The printed S1 and S4 cells differ by 1.0; the report works from unrounded scores.) It takes 0.01, “the smallest tested penalty at which every domain keeps a positive weight”. If mid-training’s penalty has the form above, any positive would meet that test, and 0.01 is simply the smallest non-zero value tried.
That budget is one where the report says mixes don’t separate after SFT, so this is a failure to detect. Taking S3, the like-for-like solve, the SFT range is 2.1, close to what the noise model can resolve, and the checkpoint averages have an interior peak (S4 61.1, S3 62.9, S1 61.8), against a yardstick of 0.6 for near-identical runs. The fit’s own optimum, S1, doesn’t win. That’s one path of single runs, and it hints that some penalty helped (our reading).
The final mix also depends on the prior. Aleph Alpha picked C3 from 400B proxy runs, in which C3 led C2 by 3.5 points after SFT. With C3 as prior, the solve takes German web from 18% to 2.1%, and the two web domains end at 3.4% between them. Olmix recommends , saying “noise from regression fitting and proxy-to-target model transfer makes moderate regularization beneficial”. Kolibri uses 0.05 in pre-training and 0.01 in mid-training, and these don’t compare: has the objective’s units, and here those are BPB in one case and a benchmark score in the other. Poolside publishes neither its nor what’s in its prior.
Solved share 0.49; true score 0.077 BPB against the true best 0.076 at 0.58.
Over the last 20 swarms at these settings: 0.49 to 0.61.
Dots: simulated proxy scores. Dashed: the target's true score. Solid: the log-linear laws fitted to the dots, summed. Ticks in the strip: the solved mix in each of the last 20 swarms.
The ideas are older than these papers, none of which cite the following (our mapping). A score as a function of shares on a simplex is a mixture experiment (Scheffé, 1958), and fitting it is a response surface (Box and Wilson, 1951). An expert merge is one round of federated averaging (McMahan et al., 2017), and its second-order gap is the leading term of what that literature calls client drift (Karimireddy et al., 2020).
Repetition
In Olmix’s example, a mix with 40% code when code is 5% of the data means oversampling it eight times. A solver gives tokens to the sources that score best, and those tend to be small, which is why runs limit passes.
The usual basis is Muennighoff et al. (2023): under fixed compute, up to 4 epochs of repeated data “yields negligible changes to loss compared to having unique data”. They model the value of epochs as fresh epochs, with a fitted . Four epochs are worth about 3.73, 16 about 10.6, and the limit is about 16.4 (computed), a limit the paper warns misses failing runs at 44 epochs, where the real loss is worse. Those are GPT-2-style models up to 8.7B parameters on C4, a web corpus. They say “we always repeat the entire available data rather than subsets of it”, and the downstream results are natural-language tasks. Using four epochs per source within a mix extends that result (our reading). They also expect that repeating earlier does less damage than later, while Kolibri’s cap lets late stages repeat.
Per-source evidence is thin. Feng et al. downsample every source by one fixed factor, so a blend keeps its epoch counts on a small run, and vary epochs directly. Math at 1, 4, 8 and 12 epochs gives averages of 51.49, 51.92, 52.28 and 51.63, with no intervals, on 8B at 1T. They recommend 6 epochs for high-quality crawl and 8 for math and task data.
A source with tokens at share of a budget is read times, so a limit of passes means . In Kolibri’s pre-training and T, so the 80B of Specialized domains can take at most 1.2% (computed). In mid-training, a dataset’s draws across both stages stop at four epochs, so its mid-training cap is four passes minus what pre-training used. In the solve the cap applies to whole domains. A separate step, a waterfill, then works inside each domain: each dataset gets its share, one above its cap is pinned there, and the excess goes to the others in proportion. In symbols, for requested shares , the same form as the capped tilt above.
Domain's budget 80B of 315B the caps allow: 1 of 5 datasets pinned.
The slider stops at the sum of the caps. Beyond it, the domain receives less than its weight.
Small runs read too few tokens for repeats to appear. At 3B tokens from a 29.3T pool, or at Poolside’s 60B, almost nothing can repeat, so a fitted coefficient says what a source is worth at about zero passes, and the solve applies it at up to three. Aleph Alpha says it directly: “At the 400B and 90B budgets no repetition cap binds.” In pre-training, the size prior and caps account for repeats. In mid-training the prior is a hand mix with no sizes in it, so only the caps do.
Aleph Alpha prepared 5.2T tokens for mid-training, with seven datasets pinned, and stopped at 3.44T, so those datasets got about two thirds of their allowance (3.44/5.2). Across the 3.44T, STEM QA averages about 2.0 passes (25.4% of 3.44T, over 431B) and German instruction 1.6 (1.9%, over 41B). Those are domain means for that stage alone, and they hide pinned datasets and passes from pre-training.
Aleph Alpha’s blog gives an example of concentration inside a domain. German text is about 4.3T seen in pre-training, and “each German token was seen 1.8 times on average”, so about 2.4T unique (computed). The German domains hold about 6.85T, so if those figures align, the run read about a third of what’s there, repeated. (The blog’s 21.3% is the report’s too; the card’s table gives its German domains 23.9%, and no source explains the gap.)
Poolside names repetition as one of two bottlenecks on M.1, with allocation the other, and answers with wider web filters, synthetic rephrasing at scale, and AutoMixer. It publishes no pass count for pre-training. DeepSeek filters out model-generated content “with limited information gain”, such as low-quality machine translation, calling it “implicit duplication” that “may become detrimental over long training horizons”. That points the other way from Poolside’s rephrasing, though the two describe different data (our reading), and the synthetic data article takes this up. Aleph Alpha is the only one of 16 labs that we found publishing a cap.
Gains and losses from fitted mixes
Most of what’s published is one four-way comparison, on Kolibri’s 11.9B proxy at 400B tokens: a run continuing pre-training, hand mixes C2 and C3, and the production mix.
After SFT, production leads the continuing run by 11.5 points, and the report splits that: 4.3 to C2, 3.5 to C3, 3.8 to production. Only the 3.8 compares a fit with a hand mix. The report’s noise model resolves a gap of 2.6 here and puts this one at . Three limits apply. Each mix has one base run, so that noise covers fine-tuning seeds and evaluation only. Half-widths of 1.9 and 1.8 give a standard error of 1.34, and a run-to-run standard deviation of about 1.0 per run would bring 3.8 down to the 5% line (computed). Nothing measures that at 400B. At 84B the near-identical pairs (S2 and S3, S5 and S6, L1 and L2, L3 and L4) differ by 0.4 to 1.2 after SFT, no more than eval noise alone predicts, though the report doesn’t state their seeds. C3 was picked on these same 400B runs. And production’s weights came after a hand edit.
The first step, the 4.3, crosses more than a change of mix: the continuing run stays at 16k and the others go to 64k, and the mid-training pool is decontaminated while the pre-training pool isn’t. The pools differ in how much benchmark text they contain, which the filtering article covers. In production pre-training “22 % to 95 % of the completions are recited” on HumanEval, and English pass@1 falls from 80.0 to 70.0 within 0.3T of mid-training as that recitation drops. The report therefore gives the aggregate without HumanEval too. Over the full model, from the M4 at the end of pre-training to the M4 at the end of mid-training, the aggregate rises 2.8, or 2.9 without HumanEval, and these cross the length change as well. The report prints no full-scale run of this model on another mix to compare with.
The losses need the same split. TriviaQA ends 8.4 points below continuing on the production mix (the report’s figure), and the hand mixes end 6.2 and about 8.4 below it, so against its own prior the fit costs about 0.1. These are gaps to a run that kept reading other data, and say nothing about forgetting by themselves. Forgetting needs a before and after, which the full model gives: M4 to M4, TriviaQA falls 6.8, German HellaSwag 5.6 and HellaSwag 2.6, while MMLU-Pro rises 2.9 and ARC 1.2. The two knowledge means fall only 1.0 in English and 1.4 in German, with single benchmarks inside them falling further. The report says “the cause of the TriviaQA loss remains open”, pointing to the difference in sequence length. Memorised text could play a part too (our reading), though at natural dilution, on the proxy, the report found contamination “not measurable”.
German HellaSwag is the loss that does separate the fit from its prior. “At 400B, the HellaSwag (DE) score of the production mix lies 5.5 points below that of C3, a cost of 0.09 points in the aggregate score.” It rises with German web across the 84B runs, from 65.7 at 0% to 79.4 at 19%, though German web also moves against code in those sweeps, and which German data is used matters too. The 0.09 is the benchmark’s weight: 0.2 for German, times 1/3 for knowledge, times 1/4 for four German knowledge benchmarks, is 1/60, and 5.5/60 is 0.09 (our derivation). A gain of 0.69 on one English math or code benchmark, with weight 1/7.5, pays for it. Hence the report’s conclusion: “Preserving a capability thus requires its own term or constraint in the objective.” The benchmark was in the objective all along, at a low weight. A domain with no path into the objective is the other case, and Specialized domains show it.
Poolside reports relative changes on a small test against its manual prior. Commonsense falls on all four benchmarks, which it calls “not surprising given the optimization objectives we chose”, and its optimisation benchmarks are “focused on coding”, though they include GSM8K and MMLU. XS.2’s mix has 30.6% raw code and 25.2% web, and no full-scale effect of the mix is published. ARC moves opposite ways at the two labs, which is a reason not to speak of one “commonsense” loss. Olmix likewise reports gains in BPB: its abstract’s “11.6%”, for reusing a mix as datasets change, isn’t accuracy.
Small budgets can hide differences. Kolibri’s mixes “separate after SFT at the larger proxy budget only”. At 84B, 14 of 120 pairs among 16 fine-tuned runs differ beyond noise, against 6 expected at the 5% level. The report estimates two thirds of those scores’ variance as noise. With 16 runs that share could lie anywhere from about a quarter to all of it, and it counts fine-tuning seeds only (computed). The pairs also share runs, so 14 is less surprising than it looks: with no real differences, about one draw in ten reaches 14 (computed by simulation). At 400B every adjacent pair differs. On those four runs the checkpoint average “appears to understate” what fine-tuning shows, though it’s a different benchmark set.
Objectives differ too, and no source compares them. Aleph Alpha fits BPB in pre-training and benchmark scores in mid-training. Poolside uses benchmark scores. Olmix uses BPB on task text, and MergeMix uses scores because lower perplexity does “not reliably translate”. Aleph Alpha’s upper-bound test shows that both can move with memorised text: adding removed documents back, 8.2% of the tokens, “inflates HumanEval by 42 points, MMLU by 29 and TriviaQA by 28 at the same step, halves the bits per byte on the benchmark text”.
Moving between mixes
A run can switch mixes at once or blend them. Aleph Alpha blends, taking the old mix’s share down linearly from 1 to 0 over the first B of the 3.44T tokens, so the old mix supplies 50B. Those tokens come from beyond what the 20T run read, so they’re new. The report cites prior work: “retaining a share of the previous data limits the forgetting” that a shift causes.
In the report’s one test, on the 84B proxy, mix C1 through the transition has a checkpoint average of 61.1, and switched directly 61.6. Without HumanEval they are 59.7 and 61.0. The test can’t settle this question. That score moves mostly with math and code, with the knowledge mean spanning 1.2 points against 5.8 for math, so it’s a poor measure of forgetting, and any window means reading less of the new mix before scoring. The report doesn’t say what window the 84B run used, the transition run had no fine-tune, and C1 sits closer to the pre-training mix than production does. The report draws no conclusion and keeps the transition.
Rehearsal, keeping some old data in the later stage, appears at other labs. Mistral tunes it jointly with the learning rate and settles on “1/5 of the peak pretraining LR and 1/10 of pretraining rehearsal”, and Nemotron 3.5 Super has 20% replay in its mid-training blend. The mid-training article covers both.
Packing
At a sequence length of 16k, a 200,000-token document fills 13 sequences. Packing turns a mix’s tokens into fixed-length sequences, and a poor method can leave a few optimiser steps dominated by that one document, even when the mix is broad. The simple method concatenates documents and splits at every tokens, with two problems that Aleph Alpha’s report names. “Almost every sequence boundary falls inside a document”, so short documents get cut for no reason. And consecutive pieces of a long document land in adjacent microbatches (the parts of a batch on one GPU), so “for a few optimiser steps the gradient is dominated by one document, one domain and one writing style”. Shuffling documents first doesn’t help, since each document’s pieces still come out back to back.
| concatenate | buffer | + packer | |
|---|---|---|---|
| Needless cuts | 16 | 24 | 0 |
| Longest run of rows dominated by one document | 6 | 2 | 2 |
| Padding | 0.0% | 0.0% | 5.9% |
Each row is one sequence, each colour one document, and strong coral is the long document. A tick at the end of a piece marks a cut. A bar at the left marks a row where one document holds more than half. Hatched cells are padding: this toy pads where Kolibri's packer overfills a sequence and drops the excess.
Best-fit packing (Ding et al., 2024) treats the first problem as bin packing, offline: put each piece in the fullest sequence where it fits, longest first. A cut then happens only for a document longer than , and sequences hardly increase (under 0.01%). Their packed arm also masks attention across documents, so tokens can’t attend to an unrelated neighbour, and the concatenated baseline doesn’t. On a 13B model at 2k, reading comprehension rises from 46.71% to 48.92% in the combined comparison, and from 47.92% against a baseline that also has the mask, so about 1.0 point belongs to packing itself.
For the second problem there’s the Random Sequential Document Buffer (RSDB) of Arcee’s Trinity report, a model outside this site’s map. It keeps thousands of open documents and reads the next chunk of a random one, so the pieces of one document spread across many steps. A small internal test shows batch heterogeneity (largest microbatch loss minus the mean) down 46%, with no benchmark.
Aleph Alpha combines them online for Kolibri’s base stages: an RSDB of 4,096 documents per worker, then a packer that fills each sequence from 1,024 pending chunks and drops tokens past the end when no subset fits, at a rate it doesn’t print. It reports “a smoother training loss”, with no number.
SFT keeps conversations whole, which leaves padding. Next-k-Fit (Johnson, 1974) holds open bins, and more bins take more memory and start-up time. At sequence length 32k, raising from 1 to 1,024 lowers padding from 14.8% to 0.6%, and Kolibri’s run at and 256k has 0.001%. NVIDIA does length-aware best-fit for SFT. DeepSeek pre-splits very long documents in pre-training and reaches “a padding rate of at most 10⁻⁴” on V4.1-Flash. Labs split in one stage and keep whole in another because the data differ (our reading; no source says so).
Eight of 16 labs mention packing in training: Aleph Alpha, DeepSeek, NVIDIA, Poolside and Liquid for fine-tuning or base stages, Mistral and Xiaomi for RL, and Z.ai for building synthetic long-context data. Three name a fit rule (Aleph Alpha, DeepSeek, NVIDIA). None publishes a benchmark change. Poolside reports a cost in a mixture-of-experts model: on M.1, about 5% of pre-training tokens were padding, and “all padding tokens in a batch are routed to the same expert at the same time, potentially saturating that expert”.
Sidebar: the tokenizer sets what a token is
Every budget in this article is in tokens, and a tokenizer sets how much text a token holds. On one German web corpus, Aleph Alpha’s figure of twelve tokenizers runs from 3.28 bytes per token (Kimi K3’s) to 4.90 (Kolibri’s), and on English from 4.16 to 4.67. So 1T tokens isn’t the same amount of text at two labs. Vocabulary size doesn’t settle it: Gemma’s 262,144 entries get 4.13 on the same German, and Kolibri’s 128,000 get 4.90.
Against GPT-5’s, the best of the nine other tokenizers on German at 4.35, Kolibri’s needs 11.2% fewer tokens there, and GPT-5’s wins on English, 4.67 against 4.58. The method, UniBPE, keeps the greedy merging of byte pair encoding (BPE) and scores merges by a different loss. Plain BPE trained on the same corpus with the same vocabulary reaches 4.89 and 4.59, so the compression seems to come mostly from training on that corpus (our inference). The report credits UniBPE with morpheme boundaries and better long-context recall. Its card prints 4.7 and 4.2, with no explanation of the gap.
Three of 16 labs state that a tokenizer was trained on their own data (Aleph Alpha, Liquid’s LFM2, Poolside), and four reuse an in-house one (Google, OpenAI, DeepSeek, Qwen’s Omni model).
The methods compared
| Method | Answers | Decided from | Costs | Best evidence, and its scope |
|---|---|---|---|---|
| Stage shift | Good data is scarce | Mostly unstated | Earlier skills can fade; length often changes too | Feng: +1.74 points over random order, 8B at 1T |
| Proxy swarm | Too many sources to set by hand | Small fresh models | Gives up what the objective weights lightly; noisy fit; scale gap | Olmix: 12.2% better average BPB than natural, 1B at 100B. Laguna: Table 3, relative, 3B at 1.5T |
| Expert merges | Runs from a base are costly | Weights from a shared base | May not rank; only first-order, at equal length | Wang, mid-training: 0.92 over 12 mixes. Kolibri, SFT: 0.77 to 0.94 with 2 or 3 specialists, −0.09 over 6 with 20. None printed for Kolibri mid-training |
| KL penalty | Solver empties sources | People: a prior and a | Result rests on both | Kolibri: 4 of 9 domains at 0 without it; SFT unresolved at 84B |
| Fit against hand prior | Does fitting beat judgement? | 400B proxy runs | −5.5 German HellaSwag, on one run per mix | Kolibri: +3.8 after SFT, against the prior it was built from |
| Repetition cap | Solver over-weights small sources | A count of passes | Basis is whole-corpus, at GPT-2 scale | Muennighoff: 4 epochs, 8.7B on C4. Feng: per source, 8B at 1T |
| Transition | Abrupt shift may cause forgetting | Prior work | Tokens not on the new mix | Kolibri: 61.1 against 61.6 at 84B, not a test of forgetting |
| Packing | Cuts, and runs of one document | Padding, loss smoothness | Buffer memory, start-up time, dropped tokens | Ding: reading +1.0 point net of the mask, 13B |
Where the models in the map sit
Token counts below aren’t like for like: some count seen tokens and others a corpus, with or without images, and some totals include the later stage. Each row names the model whose report states it.
| Lab | Later base stage on different data | Stated content of the later stage | How the mix was set |
|---|---|---|---|
| Aleph Alpha (Kolibri) | 20T, then 3.44T, then 200B | Math, code, reasoning, instruction. Then long documents and OCR’d PDFs | Fitted. Pre-training: KL to a flattened size prior. Mid-training: KL to a hand mix, then a hand edit |
| Poolside (Laguna XS.2) | Partial: more than 30T, “stage-1 mix” undefined. It files 200B long-context and about 60B “mid-training” under post-training | Mid-training 40/30/30 reasoning, code, chat | Pre-training fitted on XS.2, manual on M.1. Mid-training tuned, no method |
| NVIDIA (Nemotron 3 Ultra) | 15T + 5T inside pre-training | Quality tiers | Unstated |
| Mistral (Large 4) | 65% + 35% inside pre-training, by design, unit unstated | More code and science | Unstated |
| Liquid (LFM3) | 22T including 3.5T last | Curated, agentic traces | Unstated |
| Xiaomi (MiMo-V2-Flash) | 22T + 4T + 1T | Code up, about 5% synthetic reasoning | Unstated |
| Z.ai (GLM-5) | 27T + 1T + 500B + 50B | Long documents, agent trajectories | Unstated |
| Moonshot (K3) | Partial: long documents and videos upsampled in cooldown | “Ablation studies on smaller models”, no method | |
| DeepSeek | Partial: agentic data in V4’s mid-training | “Optimize the data mixture” (V4.1), no method | |
| Qwen (3.5-Omni) | Partial: more long audio and video in its last pre-training phase | Unstated | |
| Google, OpenAI, Thinking Machines, Tencent, MiniMax, StepFun |
A blank means searched and not found. Poolside’s “mid-training” is at most 0.2% of its pre-training tokens and Aleph Alpha’s is 17% (computed), so the label doesn’t say how much of a run it covers.
Easy to get wrong
- “Optimised” doesn’t mean tested at scale. Only Kolibri prints a full-scale chain of scores, for the mix that ran. Neither lab reports a same-model full-scale run on a second mix. Each has an earlier model on another mix (Kolibri Origin, Laguna M.1), at a different size and budget.
- Most of Kolibri’s 11.5-point lead belongs to leaving the pre-training recipe and to hand tuning. The fit’s part is 3.8.
- A domain at 0% may only have no benchmark that rewards it, as with Specialized domains, so the solver’s weights reflect the objective, more than the data.
- The >0.9 Spearman for merges is Wang’s, over 12 mixes in four domains. Aleph Alpha’s own figure at 20 clusters is −0.09, over six, and it prints none for mid-training.
- Muennighoff’s four epochs come from repeating whole corpora on natural-language tasks. Kolibri uses 3 per source in one stage and 4 per dataset across two.
- Poolside’s +43% and Kolibri’s +3.8 points differ in unit, baseline, model and budget, so they can’t be compared.
Check yourself
-
A source holds 500B tokens. With Kolibri’s pre-training limit of 3 passes at T, what’s the largest share it can take? How many passes would a 20% share mean?
Answer
. A 20% share is 4T tokens, which is 8 passes over 500B. -
Using , why can’t a domain get exactly 0% at , assuming mid-training’s penalty has this form? And why can the solve give 0% with no penalty?
Answer
The exponential is positive at any finite , so a domain with keeps some share, even one small enough to round to nothing. With no penalty, nothing pushes a share off the boundary, and the fit alone can leave a domain at zero when its slope is worse than the slopes of the domains in use. -
German HellaSwag has weight 1/60 in Kolibri’s aggregate, and an English math benchmark has weight 1/7.5. What does a 5.5-point drop on the first cost in the aggregate, and how many points on the second offset it?
Answer
, and points. A mix with that gain scores the same on the sum, so a skill you want to keep needs its own term or a constraint. -
Why does the argument for merging need a shared start and a short horizon, and what doesn’t it cover?
Answer
With a shared start, gradients are taken at one point and a mix of them equals the gradient of the mixed loss. The approximation is first order, so the second-order gap, of size , must stay small. It compares runs of equal length, so it doesn’t cover a merge standing in for a longer run: that the ordering survives is an empirical claim. -
Why is a Spearman of −0.09 over six weightings not evidence that merges never rank?
Answer
Six ranks give an interval of about −0.84 to 0.78, each trained score is one run, and the sweep changed two things at once (number of specialists, length of training). It rules out agreement near Wang’s 0.92 in that setting, and that’s all.
In the map
Methods
- Two-phase curriculum 1 source
- Curated higher-quality mid-training data 3 sources
- AutoMixer 1 source
- KL-regularized data-mixture optimization 2 sources
- Proxy-model mixture selection using fitted mixing laws 1 source
- Log-linear proxy model for predicting benchmark BPB from data mixture 1 source
- Prior-constrained Dirichlet mixture exploration 1 source
- Empirical pre- and mid-training data-mix selection 1 source
- MergeMix 2 sources
- Per-source repetition caps in data-mixture optimization 1 source
- Four-epoch dataset repetition cap 1 source
- Water-filling redistribution under repetition caps 1 source
- Linear interpolation from pre-training mix to mid-training mix 1 source
- Length-aware best-fit packing 2 sources
- Cross-source document packing 1 source
- Random Sequential Document Buffer (RSDB) 1 source
- Next-k-Fit 1 source
- UniBPE 3 sources
Taxonomy nodes
- data mixture & curriculum 117 methods
- sequence packing 27 methods
- tokenization 42 methods
Further reading
- Kolibri report, §2.3.2.4, §2.3.3, §3.1.3, C.1, D.2 and H.3: https://aleph-alpha.com/downloads/tech-report.pdf. Every mix, Table 40, the noise model and the packer.
- Laguna report, §3.2 and §3.2.3: https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf. Why manual mixes stopped working, and AutoMixer.
- Nemotron 3 Ultra report §2.3.6 and Mistral Large 4 report §2.2. The two curricula inside pre-training, with their figures.
- Hu et al., MiniCPM, arXiv:2404.06395, and Feng et al., Two-Phase Pretraining, arXiv:2412.15285. The early tests of staging, and per-source epochs.
- Chen et al., Olmix, arXiv:2602.12237. Seven design choices of the swarm schema tested, and reusing a mix when the datasets change.
- Liu et al., RegMix, arXiv:2407.01492, and Ye et al., Data Mixing Laws, arXiv:2403.16952. Where the regression and the log-linear law come from.
- Wang et al., MergeMix, arXiv:2601.17858. The second-order argument, in Appendix B.
- Muennighoff et al., arXiv:2305.16264. The repetition law, with its limits in the appendices.
- Ding et al., Fewer Truncations, arXiv:2404.10830, and Singh et al., Arcee Trinity Large, arXiv:2602.17004. Best-fit packing and the document buffer.
