Open Models Almanac

Data · D3

Synthetic data: what models write for other models

A generator writes what you can't crawl. It passes on its errors and biases, so what counts is whether anything can catch them

Illustration of a writing desk with one old open book, a fan of fresh pages copied from it in different shapes of handwriting strokes, and a small stack of pages set aside with coral marks.

Kolibri, Aleph Alpha’s 78B-parameter model, reads 20T tokens in pre-training, and about 4.8T of those were written by other language models. German shows why. Open German datasets came to 390B tokens after deduplication and filtering (its blog), and the run wanted at least 4T German tokens seen. So Aleph Alpha curated its own German crawl, then reused about 1.1T unique tokens of web pages that a model had rewritten as encyclopedia entries, dialogues and lessons for an earlier project.

DeepSeek V4.1 filters “model-generated content with limited information gain” out of its pre-training text, as “implicit duplication”, without saying whether it means text found on the web or made in-house. The same report’s post-training section credits synthesized data and environments with “essentially all of the observed gains”, an attribution with no number or ablation behind it.

Both statements fit one report because the two stages differ. In pre-training, generated text sits beside trillions of organic tokens, and only another model reading the source can tell you that a rewrite got a fact wrong, with blind spots of its own. In post-training, nobody publishes millions of worked reasoning traces or coding tasks with a grader, and for some of that data a test can catch a wrong answer.

The sections run from where organic text is most plentiful to where there is none: rephrasing, translation, long-context data, answers for fine-tuning, then tasks. At each step, ask what could catch a generator’s mistake.

Background

The primer covers the stages of a run. Pre-training is the long first stage on broad text, followed by shorter base-model stages, mid-training and long-context extension. Post-training means supervised fine-tuning (SFT), which trains on conversations, and reinforcement learning (RL), which trains on tasks with a reward. Tokens come in kinds: seen tokens count repeats, a pool is what a source holds, unique tokens are the distinct ones read, packed tokens fill the final sequences, and loss tokens are those the loss has targets on. A proxy is a small model standing in for a run nobody can repeat, and an ablation changes one piece of a recipe to see what it did.

A generator or teacher is a model whose output becomes training data. (Labs also say “teacher” for a model that supervises a student as it generates, which is a different operation.) Rephrasing rewrites a given document in another form, and other pipelines generate text from a prompt or a short seed with no document behind it. A reasoning trace is the text a model writes before it answers. A verifier is the code that decides whether a task was solved, and its verdict often becomes the reward.

Modeling code records none of this, so we read technical reports first, then cards and official blogs, and name outside papers where we use them. We count labs out of the 16 on this site, and a blank means we searched and found nothing, which doesn’t mean a lab doesn’t do it. Most reports give a recipe without a size or an effect, so we mention a missing number only where it changes how to read a claim. Reports describe what shipped, and none says that a synthetic pipeline it kept added nothing overall.

Figure 1.Where generated data enters and leaves Kolibri's run, as the report states it. Stage lengths aren't to scale. The report's reason for the mid-training rule is that rephrased and translated German "read less naturally" on manual inspection.

Pre-training rephrasing

Labs give two reasons for rewriting. A run needs more tokens than good text supplies without repeats: on M.1, Poolside’s earlier model, it found “excessive repetition in high-value subsets”, and for Laguna XS.2 it “expanded the amount of usable training signal through targeted synthetic rephrasing at scale”. Moonshot’s K3 follows a recipe from Kimi K2, whose report (outside our corpus) says one epoch “is insufficient for comprehensive knowledge absorption”, and more bring “diminishing returns”. The other reason is scarcity in a language, as with Kolibri’s German.

Nemotron-CC (Su et al., 2024) set the pattern most labs cite. A prompt turns each seed document into a new form, with no instruction to add knowledge: the approach “does not utilize the language model as a knowledge bank”. It has five forms, a Wikipedia-style rewrite, question-answer pairs appended to the document, a distillation, an extraction of knowledge and a knowledge list. Kolibri’s English data uses a “Nemotron-CC-style recipe” with Gemma-4-26B-A4B, and its German data uses three styles from Mistral-Nemo-Instruct-2407. Moonshot adds “fidelity verification against the source documents”, which it doesn’t describe further.

Figure 2.An invented example of three of Nemotron-CC's five forms. Exact and fuzzy deduplication see these as different documents, by design.

Generators here tend to be small, because the volume is enormous: 12B in Nemotron-CC, 26B at Kolibri. That may do. Outside studies find little or no benefit from a larger generator above 1B (Niklaus et al., 2026) or 3B (BeyondWeb, 2025), and Niklaus et al. find “the choice of rephrasing prompt is the dominant factor”, which is the part labs print least of. Kolibri names its templates, Ultra states no recipe for syn-crawl (NVIDIA’s label for rephrased crawl), and Moonshot gives adjectives.

Five of the 16 labs state a share or a recipe for generated text in the broad pre-training mix. The numbers don’t compare, since each is a share of something else.

ModelStated shareShare of what
Kolibriabout 24%20T seen, both languages, rephrases and translations
Nemotron 3 Ultrasyn-crawl 32.7% then 26.2%; with “synthetic SFT-style data”, at least 47.4% then 43.3% (computed from Fig. 4)each phase’s sampling mix, 15T then 5T
Laguna XS.2about 13%the mix across all stages, from about 4.4T generated
MiMo-V2-Flashabout 5%, reasoning dataa 4T stage it calls mid-training, inside the 27T its abstract calls pre-training; unit unstated
Kimi K3no share printedrephrases knowledge and mathematics

No lab rejects synthetic pre-training text outright. Three state limits:

  • DeepSeek’s V4.1 filter covers “outputs from less capable models and low-quality machine-translated text”, and the same paragraph explores “model-in-the-loop data iteration approaches as a foundation for future large-scale synthetic data”. A separate report, V4, removes “batched auto-generated and templated content” from web data.
  • Z.ai’s GLM-5 report says pipelines “strictly avoid the use of synthetic, AI-generated, or template-based data”, in its paragraph on math and science. It builds synthetic data for later stages.
  • MiniMax’s M3 blog says “interleaved data scales more easily than synthetic data”. From its heading, interleaved seems to mean mixed image and text documents. That compares ease of scaling, and doesn’t say synthetic data was dropped.

Of the other eight, Mistral states generated data only in mid-training, and Thinking Machines’ card says “synthetically generated or augmented” with no stage. The other six say nothing on broad text.

What a rephrase can add

DeepSeek says the low-gain text it filters “largely reformulates existing information”. Poolside says of rephrasing that it expands “usable training signal”. They aren’t speaking of quite the same text, and each claim can be made precise (our derivation, in no report).

Write I(X;Y)I(X;Y) for how much knowing YY tells you about XX. Let SS be the world the text describes, DD a document, MM the generator, which was trained on text about SS and so depends on it too, and RR a rephrase. Once you hold DD and MM, the world affects RR through nothing else, so S→(D,M)→RS \to (D, M) \to R is a Markov chain, and what the rephrase adds to the document is bounded by what the generator knows beyond it:

I(S;R∣D)≤I(S;M∣D).I(S; R \mid D) \le I(S; M \mid D).

A verbatim repeat adds zero. If the generator knows nothing more about SS than the document does, a rephrase adds zero too, and that’s the sense in which it’s duplication. With the whole corpus in place of DD, a generator trained on the same web has little left to add. Mutual information doesn’t separate true from false, so this bound says nothing about errors. It also describes an ideal learner, one that already knows how wordings relate.

Training doesn’t start out knowing that, which is where Poolside’s claim has room. Take a linear toy with one feature for a fact and one for each wording, σ2\sigma^2 being the strength of wording features against the fact feature. Trained from zero weights to zero loss on KK wordings of a fact, the model’s margin on a wording it hasn’t seen is K/(K+σ2)K/(K+\sigma^2), against 1/(1+σ2)1/(1+\sigma^2) for KK verbatim repeats, and more steps don’t close that gap. A model that memorises strings has to see several wordings before it picks up the fact.

Errors change the answer, and how they correlate matters more than their rate. Keep the original among the KK texts. If each other wording is wrong independently with probability ε\varepsilon, rephrasing stays ahead of repeating while ε<σ2/(2(1+σ2))\varepsilon < \sigma^2 / (2(1+\sigma^2)), at any KK. If the generator gets this fact wrong every time, the margin is (2−K)/(K+σ2)(2-K)/(K+\sigma^2), and each further rephrase makes it worse. We found no report that measures how correlated a generator’s errors are across rephrases.

Margin on an unseen wording, from −1 to 1. The original text is always correct.

Threshold σ²/(2(1+σ²)) at σ² = 4: ε below 40.0%

At ε = 10.0%, independent errors stay above repeats

Figure 3.A toy, with no real data. It has one fact and no new knowledge, so it says nothing about rephrases against fresh text, only against repeats. It predicts no gain for a learner that doesn't memorise wordings (small σ²).

What’s measured

One report on a shipped model isolates pre-training rephrasing, and it’s narrow. On an early checkpoint of Kimi K2, SimpleQA scores 23.76 on raw wiki-text at 10 epochs, 27.39 with one rephrasing, and 28.94 with ten rephrasings at one epoch each. The recipe it shipped rephrases each corpus at most twice, and it calls synthetic data for scaling “an active area of investigation”.

Su et al.’s Table 10 tests the recipe most labs copy, on four 8B models at 1T tokens, each a single run.

BlendARC-EMMLUThe paper’s average
Low-quality documents67.748.252.5
Low-quality documents, rephrased71.347.154.0
High-quality documents, eight passes74.253.455.8
Four of those passes swapped for rephrases76.753.656.7

The second pair is the alternative a lab faces when it rephrases to avoid repeats, and rephrasing gains 0.9 points (computed). The first pair gains 1.5, and MMLU, a test of knowledge, falls 1.1. The authors write that such drops “may indicate potential misinformation introduced by data synthesis”, and that they “did not verify the factual accuracy or fidelity to the original contents”.

Kang et al. (2025) train over 1,000 models of up to 3B on up to 200B tokens, against fresh raw web text, so they test something else: their high-quality rephrasing aims to be, in their words, “akin to an aggressive data filtering”. A mix of one third rephrased text reaches the same validation loss up to 5 to 10 times sooner at larger budgets, though rephrased text alone isn’t faster. Niklaus et al. put the best mix at 60% to 80% synthetic, against Kang’s roughly 30%, on different metrics.

Text generated from seeds or prompts, with no document to rewrite, looks different. Kang’s two-thirds textbook mix underperforms, while one third overtakes raw web after about 20B tokens. Phi-4’s 13B ablation models on synthetic data alone score 14.8 points below phi-3-medium on TriviaQA, and an equal share of web rewrites halves that to 7.7, while the other seven benchmarks hold or rise. NVIDIA’s Ultra tests benchmark-seeded questions in a 100B ablation on an unnamed Nemotron checkpoint: GPQA moves from 30.8 to 41.9, MMLU-Pro from 64.8 to 66.6, and math holds. Those are seeded from benchmarks’ training splits, so the gain is hard to read.

In short, Su’s one pair and K2’s one test favour rephrasing over repeats, by 0.9 points and by 3.6 to 5.2 points (computed), with no repeated runs, on models up to K2’s early checkpoint. Kang’s favours a rephrased mix over raw web, on a loss. Labs running at larger scale are extrapolating from these (our reading).

Model collapse

The studies behind the phrase “model collapse” train a model on its predecessor’s output, repeatedly. Rare events disappear from the output first. Shumailov et al. (2024) find “only early signs” in a fine-tuned OPT-125m, and Gerstgrasser et al. (2024) find, on a 9M GPT-2, that replacing data hurts where accumulating it doesn’t. None of these tests rephrasing by a different model family across several generations at scale.

The paper DeepSeek V4 cites, Zhu et al. (2024), tests one generation: small models trained from scratch on a mix of human text and text generated from prompts. Perplexity on human text rises about 5% at a quarter synthetic and about 2.3 times at all synthetic (computed over its 22 domains). Any text at share ss dilutes the target, so an ideal learner’s perplexity can rise by up to 1/(1−s)1/(1-s), and the measured rises up to three quarters stay below that (our derivation). Its task averages barely move, about 41 on GPT-2 and 33 on OLMo-237M, though that may be a floor, since guessing gets about 33 on OLMo’s six tasks and about 35 on GPT-2’s seven (our estimate from option counts). The authors conclude that “relying solely on synthetic data leads to model collapse” and propose editing human text at the token level. Kang’s rephrased mixes point another way, on another kind of text, and no head-to-head at scale exists.

What it costs

A rewrite is built to differ at the string level, so it passes exact, substring and fuzzy deduplication, and repetition caps count it as new (our reading; we found no report whose deduplication compares a rephrase with its source). That is the practical form of “implicit duplication”, and it’s why a count of unique tokens can flatter a pool.

Generated text can carry benchmark content, too, since generators learned from benchmarks and some are seeded with them. In Kolibri’s curated mid-training pool, “synthetic instruction and question-answering data holds 32 % of the inspected documents but 67 % of the removals”, at about 4.3 times the rate of everything else (computed). The method finds benchmark-shaped text most easily, so that gap may partly reflect what it can see. The filtering article covers the method, and why a null at proxy scale doesn’t carry to a full run. For SFT, Kolibri adds that “paraphrasing can still get past this filter”.

Translation

Translation is rephrasing into another language, used mostly where that language is short of text, and its output can be scored against the source. NVIDIA’s Indic pipeline has four stages for twelve priority languages: translate, score from 0 to 100 and list errors, rewrite what scores below 75 against those errors, score again, and keep at 75 or above. Gemma-4-31B-it does all four. Translation alone has a median cost of about 0.335 GPU-hours per million input tokens on a benchmark, after speed work, so the 1,358B of native-script input in those languages would come to some 455k GPU-hours (computed, rough: the report says its input counts aren’t total processing, and doesn’t say the whole corpus ran at this rate).

Figure 4.The corpus is built for release and for later models, so one stage's draw says little about its use. The 10B is a configured allocation of synthetic and natural Indic text together, with the split unstated. 327.2B of the total comes from the ten-language pipeline, with Bodhan-MT as translator.

The report names the problem: “using the same model for translation and judging may favor its own translations”. There’s a precise worry behind that. With pp the writer’s error rate on an item and mm the judge’s miss rate, the chance that an item is wrong and passes is E[p]E[m]+Cov(p,m)E[p]E[m] + \mathrm{Cov}(p,m), if the two err independently on a given item. A model judging itself catches its slips and shares its blind spots, so the covariance stays. A second model lowers it without removing it, since hard items are hard for everyone (Eckhardt and Lee, 1985, on multi-version software). Human review is “sampled”, and could estimate the miss rate on kept documents, but the report prints none.

The gains on five Indic benchmarks, 4.98 to 9.82 points against the checkpoint before that 100B stage, belong to the stage, since “they do not separate the contributions of synthetic and natural Indic text”. Filters can miss translated text: NVIDIA’s Nemotron 3 Nano (outside our corpus) found after training began that its quality classifiers rated some uninformative translated documents highly.

A judge checks whether a translation matches its source, and it can’t supply what the source never held. Aleph Alpha writes: “A corpus translated from English therefore remains, in many respects, an English-language view of the world expressed in German.” It excludes rephrased and translated German web text from mid-training because it “read less naturally”. A later proxy sweep neither confirms nor refutes that: runs on an extended pool score 2.4 to 3.1 points lower on German HellaSwag (a commonsense test), against at most 0.9 between curated runs, but no pair test after SFT clears the noise, and that pool also adds low-quality web tiers. Kolibri doesn’t translate reasoning traces at all, since doing so “leaks artifacts of the English trace”. It translates prompts, and a teacher writes a new German trace.

Translation as training data is older than these pipelines: back-translation gained 2.8 to 3.7 BLEU in machine translation (Sennrich et al., 2016). There “only the source side” is synthetic. In a language model every token is a target, so a translation error is a wrong label.

Long-context data

Long documents exist. What doesn’t exist, as K3’s report puts it, is range for free: “Length alone, however, does not confer long-range capability.” A question about the last page can be answered from the last page (our gloss). Labs respond differently by stage.

In base stages, the data trains attention at length. Moonshot permutes and concatenates documents and sub-tasks, so that tasks “can be solved only by attending to information scattered across the full 1M-token context”. Z.ai’s GLM-5 uses natural and synthetic data, with similar texts packed together and “MRCR-like data” at 200K.

In SFT, the data supplies questions that span a document. Kolibri’s recipe takes about 78k documents of 128k to 224k tokens. Qwen3.8-27B reads a document with an index of about 40 sections and writes four simulated requests, each tied to the sections it needs, and in 77.4% of conversations the requests draw on at least three of the document’s five parts. A separate call usually answers from just those sections, so the answering call needn’t handle the length, while the student reads the whole document. NVIDIA’s Nemotron 3 Super concatenates related documents up to 512K and has an LLM write questions needing “at least 4 to 7 distinct retrieval or reasoning steps”, then eight sampled traces vote on the answer.

These data can resemble what benchmarks test. RULER (Hsieh et al., 2024) “consists solely of synthetic tasks” with fixed templates: find the magic number, track a variable. AA-LCR, by contrast, has 100 human-written questions over real documents. So Aleph Alpha removes synthetic samples from the naturally long pool, one third of its base stage’s mix, “to avoid artificially favourable results on synthetic long-context benchmarks such as RULER”. NVIDIA’s Ultra says of its long-context phase, “We did not use any RULER-style data in the blend”. Both are precautions, and we found no paper that measures the inflation. A measurement would compare the gain on a templated test with the gain on a test of the same skill that the data can’t resemble.

Figure 5.Single runs on Aleph Alpha's proxy, with a 5% share of the data added. The star is p below 0.05 in the report's paired test, over questions, so it compares two fixed checkpoints and doesn't cover retraining. Paired runs can differ while their intervals overlap. The appendix says AA-LCR and LongBench Pro "fall just short of significance", where the main text says the data "improves" them. Elsewhere the report finds its proxy predicts the direction of a change in 33 of 45 cases.

The one significant gain is on RULER, though this data isn’t RULER-style. It doesn’t show inflation: variable tracking minus AA-LCR is +7.2, with an unpaired half-width of about 17 (computed), so this table can’t tell either way. AA-LCR’s +8 is a failure to detect, at the power of about 100 questions, and not evidence of no effect.

Answers for fine-tuning

Web crawls don’t hold reasoning traces or tool-use conversations, and people can’t write millions. Eight labs state that an SFT or equivalent stage uses data written by models, Qwen under another name for the stage, and Z.ai is a partial ninth. NVIDIA writes that “SFT in turn consists mostly of synthetic data”, and adds that “very long SFT stages can reduce how much the model later gains from RL”. About 30% of Poolside’s SFT tokens are single-turn trajectories (records of one attempt) from “an open-source teacher model”, and about 45% are a corpus of the same size with reasoning traces added.

Three operations recur, and only one has a check on the output itself.

Regeneration keeps a dataset’s prompts and replaces its answers using a better teacher. Aleph Alpha found “many completions of mixed quality, including wrong answers to science problems, which led to regressed benchmark performance if included in training”. This is pseudo-labelling on a large scale. Its check, where there is one, is a judge or a match against a known answer, and NVIDIA uses a length filter, removing reasoning “shorter than 5,000 tokens”.

A backfill adds reasoning to a tool-use conversation that has none, written to lead to a call already recorded. Kolibri has a test: “The reasoning is kept only if a second LLM call derives the same tool call from it alone.” That shows the reasoning determines the call. It doesn’t show that the prompt leads there, or that the call was right, and since the teacher wrote it knowing the answer, a trace that simply states the call would pass.

A simulated user writes the later turns of a conversation. Kolibri’s reason: “Open data holds few long conversations, so we generate them.” Two labs state this for SFT data, Aleph Alpha and NVIDIA, which trains only on the final assistant response, since earlier turns can be poor. MiniMax has a simulator it uses in “training and evaluation” without saying how, and Mistral uses one inside RL environments.

Regenerated and in-house: 63.8% of 335.8B pool tokens.

Open data as released isn't human-written: the report says most open datasets are generated by other models.

Both totals: 63.8% of pool tokens (31.7 + 32.1, the table's "All" row), 42.5% of loss tokens (computed). The loss view assumes each domain's loss tokens split among its segments like its tokens. Without that, the total lies between 26% and 56%.

Figure 6.The pool, from which each run draws 268B packed. "Regenerated" also covers answers completed with reasoning or translated to German. Both totals are computed. Long context is 14.8% of pool tokens and 0.6% of loss tokens, since the loss has targets only on the answers. Its forward passes still cost compute for every token.

Does a stronger teacher or a filter help?

NVIDIA’s 3.5 report says that with its older teacher, “downstream reasoning performance largely plateaued after a single pass”, and with data regenerated by Kimi K3 “performance continued to improve over multiple passes”. Its hypothesis is that K3’s “more concise and information-dense reasoning” needs “additional exposure”. The contrast crosses runs and prints no numbers: Ultra’s report says of the earlier teacher’s data that “the student has not directly seen” it, since that data trained a separate teacher.

The largest public test is OpenThoughts (Guha et al., 2025), with SFT on a 7B Qwen student from 31,600 examples per run. A small gap between teachers can be a large gap between students. QwQ-32B and DeepSeek-R1 are about level on the paper’s three math benchmarks, 89.2 and 88.1 (computed from its Table 29), yet students on their math data reach 71.6 and 62.5 to 64.8, and Phi-4-reasoning-plus scores 85.4 with a student at 49.0, so on math the order holds and the gaps widen. On filtering, “we experimented with numerous verification and answer filtering methods, and none gave significant performance improvements”. Kolibri cites this to skip a filter on German chat replies, since “it adds compute without a known benefit”. These are small students, and no lab publishes the same test at scale.

Xiaomi gives the only before-and-after we found with token counts for one teacher’s data going into a student from another family. It fine-tuned Qwen3.5-9B on 77.4B tokens from MiMo, 27.2B of them loss tokens. SWE-bench Verified barely moves, 60.0 to 61.1, and SWE-bench Pro goes from 32.0 to 44.6. The baseline is the released student, with no control on other SFT data.

Keeping a teacher’s reasoning in German

A language can be a teacher’s blind spot. Asked to reason in German, a teacher often slides into English. Aleph Alpha measured more than 5,000 completions each: with an instruction in the system prompt, 67% of traces stay German throughout, 22% switch partway and 11% start in English. Prefilling, which here means writing the start of the model’s reasoning for it (a German opener such as “Der Nutzer möchte”), gets 97%. Despite the name, reasoning-prefill distillation is offline data generation. It intervenes where a filter would select: dropping English traces at 67% would also drop whichever prompts push the teacher out of German.

Those figures describe the teacher, and no run tests the prefill on a student. Table 51 tests all German SFT data on a German math benchmark, by raising its share. From none to 26.5% of the mix, the overall German score moves −4.4 (not significant) while answers both correct and in German rise 23.5 points. With no German SFT data, about 60% of the correct answers were in German (42.3 of 70.2), and with any share all are (computed, reading the overall score as counting any language), so the dip compares different behaviour. These are single runs per share.

Tasks and verifiers

A web crawl holds no graded tasks, so here generation is close to unavoidable, and a test can check the result. Here’s the shape of one coding task.

Figure 7.A typical shape, built from what several reports describe. When a repository has no suitable tests, a model has to write the verifier as well.

Eight labs describe a method for writing tasks, StepFun does so in a blog outside our corpus, and Thinking Machines gives a clause. The rewards article covers the rewards. Here we ask how a task gets built.

“Synthetic” covers a range. Laguna’s section “Synthetic Code Environments” mines commits, and it keeps “on the order of 30–60k tasks from a raw pool of ∼236k commits”. SWE-rebench V2 (Badertdinov et al., 2026), which Kolibri builds on, keeps 32,079 of 805,598 candidate pull requests, about 4% (computed), with human tests. DeepSeek V3.2 has 4,417 tasks with both environment and prompt synthesized, out of 85,267 (computed), and MiMo-V2-Flash 50K of 320K. So counts of environments don’t compare: the 409k of Laguna S 2.1 cover agentic and non-agentic work and include purchases from vendors.

What a verifier is checked against depends first on what exists. Where a repository has tests, labs mine them. Laguna requires “the gold solution to pass the tests and an empty solution to fail them” (its check), and Mistral keeps “tests that fail before the reference solution and pass afterwards”. Mined tasks exist only where a repository has usable tests, which favours some kinds of code.

Where nothing exists, a verifier has to be written, and two designs appear, with no comparison published. In DeepSeek V3.2’s general agent pipeline, “the agent will modify the solution or verification functions until the solution’s output passes the verification”. The two agree by construction, so only a later filter tests them: “retain only instances with non-zero pass@100”, meaning a model passed at least once in a hundred tries. GLM-5.3’s verifiers “are synthesized without access to the reference solution”, for a subset of tasks, and must pass “oracle, no-op, and unsolved-state checks” (we read these as: a known answer passes, doing nothing fails, the unsolved state fails). Those pipelines “still require a meaningful amount of human-in-the-loop work”.

A third route makes tasks from tests that exist. Kolibri has GLM-5.3 inject a bug: “A bug is kept only if at least one test fails and at least half of the tests still pass, so that it is detectable but local.” That’s SFT data, from a teacher fixing the bug.

Why does verifier error matter? A verifier that errs at random, on any attempt alike, slows training without changing what’s best. With α\alpha and β\beta its false-positive and false-negative rates, expected reward is α+(1−α−β)P(solved)\alpha + (1-\alpha-\beta)P(\text{solved}). Errors tied to how the task was attempted do change it: the policy moves to whatever passes most often, so an exploit only has to pass more often than honest work, and hard tasks are the most exposed. RL searches for such errors, where an SFT dataset only contains the errors already in it. Six labs report observing reward hacking, passing the check without doing the task, during training: DeepSeek, Moonshot, Z.ai, NVIDIA, Mistral and Xiaomi. Some of these cases involve reward models and not written verifiers. Poolside reports it in evaluations run during post-training.

Labs use different checks against different flaws.

FlawCheckExample
Reference fails, or empty solution passesRun bothLaguna; GLM-5.3
Flaky tests or setupRerunSWE-rebench three times, Xiaomi V2.6 eight
No signalBand of pass ratesPoolside drops tasks “always solved or never solved”; Mistral calibrates difficulty by “solver pass rates”
Rejects valid work, accepts wrong workSolvers of varying strength, an auditMistral, to “distinguish model failures from flaws in the specification or tests”; Xiaomi attempts each task four times
Solution left behindScrub the environmentXiaomi calls leakage “a recurring failure mode”; DeepSeek V4.1’s setup agent “removes any traces”
ShortcutAn attackerMistral’s “adversarial agent”; Xiaomi’s hack agent, run “until the hack agent could no longer find a successful exploit”

Rates are scarce. Xiaomi gives a confirmed-hack share “below 2%” of trajectories after that cleanup, a floor since it counts what was confirmed. Poolside’s judge flagged over 50% of trajectories on SWE-bench evaluations as post-training went on, and below 2% after a prompt told the model not to use solutions found online. SWE-rebench’s LLM judge finds some leakage in 23.0% of 509 generated problem statements, and 2.4% explicit.

Each check also selects. What’s kept follows p(x)a(x)p(x)a(x), with a(x)a(x) the chance of passing, and nothing reweights it afterwards. A test that flips 5% of the time still passes three reruns 86% of the time (computed). A pass@100 filter keeps a task that a model solves 3% of the time with probability 95%, and keeps it just as readily if the model passes by an exploit. A bug filter keeps bugs the suite already catches, unlike many that ship. Pass rates also go stale as the student improves, and “some model solved it” removes what no model can solve yet. We found no report that prints what its filter did to difficulty. All this costs compute as well: eight traces, four attempts or a hundred rollouts per task.

Generators have habits in this job too. StepFun writes that agents “repeatedly converge on familiar task patterns” and “tend to construct easier tasks”. DeepSeek V4.1 trains its model to construct tasks, using “difficulty and correctness as reward signals”, and says “this capability remains far from perfect”. Five labs also build questions that take several search steps, from graphs or existing sets, and Z.ai and Aleph Alpha drop what a model answers without searching.

Who generates for whom

Most named external teachers are other open-weight models, and several labs on this site are each other’s sources.

Figure 8.What each lab states about its own generators, by stage. Read a cell as column uses row. It's our reading that these labs now write for each other, since no document argues it.

Datasets carry this further than models do. Kolibri regenerates answers on NVIDIA’s released Nemotron datasets and translates Toucan, a tool-use set built with Kimi-K2, GPT-OSS and Qwen3. Generator choice also follows the job. Aleph Alpha writes: “smaller models are generally faster for long-context tasks, but larger models are more suitable for generating strong reasoning traces. They also have different values and biases, which we consciously select or avoid”. Google’s and OpenAI’s own documents don’t mention that Gemma and gpt-oss write data for others.

Labs filter for two traits they expect from teachers, identity and politics. “Most open datasets are generated by other models, and responses often carry their identity”, so Kolibri drops conversations in which the assistant claims to be another model. It adds that “several of our teacher models were built in China, so they carry known political biases on sensitive topics”, and has gpt-oss-120b judge open-ended conversations for that. NVIDIA’s cards state a keyword filter for teachers’ “nationalistic narratives”.

On licences, NVIDIA’s cards say that for math, code and science they “use open source permissive models such as GPT-OSS-120B”, and Aleph Alpha’s card says “permissively-licensed LLMs”. We found no document discussing a specific teacher’s terms. Open weights don’t settle whether a model’s outputs may train another. Laguna says a pre-training token “can be traced back through deduplication, filtering, and synthesis to its source document”, which covers one lab. A teacher’s own training data is usually unknown, and a student can later become a teacher, so an error could pass through several models with nobody able to trace it (our reading; no one has measured it).

Most of this has older names. Learning from a teacher’s samples at temperature 1 has the same expected objective as distillation with soft targets at temperature 1 (Hinton et al., 2015, raise it above 1), with more variance, and at lower temperatures the target sharpens, which is a bias. Training on invented inputs labelled by a model is model compression (Buciluǎ et al., 2006). A bug filter borrows mutation testing’s mechanism (DeMillo et al., 1978), a pass-rate band is goals of intermediate difficulty (Florensa et al., 2018), and a model trained to write tasks is unsupervised environment design (Dennis et al., 2020). The scale is new, and so is generated text being the target itself.

The methods compared

The table runs from weakest check to strongest.

MethodOrganic alternativeWhat the check can catchWhat it costsBest evidence, and its scope
Generation from seedsLittle in that formRarely statedKnowledge, hallucinationPhi-4: TriviaQA −14.8, other seven level or up, 13B ablation models
RephrasingRepeats of good textJunk output; wrong facts only if checked against the source (Moonshot, no detail)Errors, no new facts unless the generator adds them, GPUsSu: +0.9 over repeats, 8B at 1T. Kang: sooner to a loss than raw web, up to 3B
Simulated usersFew long conversationsLittleGenerator’s idea of a userNone isolated
RegenerationWeak open answersA judge or a known answerTeacher errors, identity, politicsOpenThoughts: filters null at 7B; near-level teachers, students far apart
TranslationScarce native textA score, often from the same modelShared blind spots, a translated view of the worldNVIDIA: stage gains, synthetic and natural together
Long-context requestsLong documents, few spanning questionsSpread across sectionsCompute, few loss tokens, resemblance to benchmarksKolibri: four of five rise, one significant, single runs on a proxy
BackfillNoneThat the trace determines the callReasoning written in hindsightNo student test
Written tasksMined tasks, where tests existSolvable, detectable, stable, partly unhackableRollouts per task, wrong rewards, a shifted setYields and two hack rates; no comparison of verifier designs

Where the models in the map sit

Sizes aren’t like for like.

LabPre-training textTranslationLong-contextTeacher dataNamed generatorsTasks and verifiers
Aleph AlphaAbout 24% of 20T seenCurated set; traces neverRequests over sections; removed from one pool63.8% of pool tokens own or rewrittenGLM-5.2, 5.3, Qwen3.8-27B, Gemma, othersBug injection, multi-step search
NVIDIA32.7%, 26.2% syn-crawl; floor 47%, 43%Indic, 2.83T generatedConcatenation, graphs, chunked translationYes; shares not printedK3, V4 Pro, GLM-5.2, gpt-oss-120b, othersWikidata, terminal, execution checks
PoolsideAbout 13% of XS.2’s mixAbout 30% and 45% of SFT tokensUnnamed open modelMined tests, two-sided check
MoonshotRephrasing, no sharePermute and concatenateYesOwnGraph-guided search; hidden verifiers
XiaomiAbout 5% of one stageYes; MiMo into a Qwen studentOwnAudit, 8 reruns, hack agent
DeepSeekFilters limited-gain textFilters low-qualityGenerator not statedOwn V3.2 for pipelinesCo-written verifiers, inspection agent
Z.aiAvoids in math and scienceNatural and syntheticPartialUnnamedBlind verifiers, knowledge graph
MiniMaxInterleaved scales more easilySimulator only
MistralTrajectories in mid-training onlyFail-to-pass, solvers, attacker
LiquidVision and audio onlyLFM2-VLYes, including text turned into audioUnnamed open models
QwenIn-house teachersOwn
Thinking MachinesCard, no stageSmall bootstrap SFTKimi K2.5One clause
StepFunBlog outside corpus
Tencent, Google, OpenAI

A blank means searched and not found. Eleven labs state generated data in post-training in our corpus, StepFun in a blog outside it, and MiniMax a simulator (computed from this table).

Easy to get wrong

  • “DeepSeek rejects synthetic data.” It filters limited-gain model text in pre-training and explores synthesis in the same paragraph.
  • “Model collapse” as a verdict on rephrasing. The studies test recursive training on small models, or one generation of text generated from prompts.
  • Kolibri as “synthetic for pre-training, organic later.” The stated exclusion is German rephrased and translated web text. Synthetic instruction and QA data is the curated pool’s largest class by documents, at 32%, and the report doesn’t say whether English rephrases stay.
  • 2.83T as Super’s Indic data. It’s the corpus. Super’s stage has at most 10B of Indic text, natural included.
  • 63.8% as the synthetic share of Kolibri’s SFT. It’s what Aleph Alpha wrote or rewrote, as a share of pool tokens, and much of the remainder is model-written elsewhere.
  • “Synthetic environments” as written by agents. Many are mined from commits, with only the setup generated.

Check yourself

  1. A lab has 500B of good text and wants 2T seen tokens from it. How many passes is that? If it swaps two of the passes for two different rephrases, about how many tokens of new strings does it get, and how many new facts, if the generator knows nothing beyond the documents?

    AnswerFour passes. About 1T of new strings, if each rephrase is about as long as its source. New facts: none, by the assumption, and the bound on I(S;R∣D)I(S;R \mid D) says that assumption is what matters. Some statements may also be wrong. Whether a model learns more from these than from repeats is what Su et al. test.
  2. Kolibri’s long-context domain has 49.8B tokens and 0.9B loss tokens. What share of its tokens does the loss have targets on? Why do its share of the pool and its share of loss tokens tell different stories?

    Answer0.9/49.8≈1.8%0.9/49.8 \approx 1.8\%. So it’s 14.8% of pool tokens and 0.6% of loss tokens. The pool share tracks the compute of reading those tokens, and the loss share tracks how many targets they supply.
  3. A verifier has false-positive rate 0.2 and false-negative rate 0.2, at random across attempts. What’s the expected reward for a policy that solves a task half the time, and for one that never does? Then suppose an exploit passes 70% of the time and honest work passes 60%. What does training favour?

    Answer0.2+0.6×0.5=0.50.2 + 0.6 \times 0.5 = 0.5, and 0.20.2. The ordering is preserved, with a smaller gap, so training slows. In the second case the policy moves to the exploit, since the verifier’s error depends on the attempt.
  4. Why does V3.2’s general agent pipeline need a pass@100 filter when its solution and verifier already agree? What does the filter still miss?

    AnswerThey agree by construction, since the agent edits either until they match, so agreement can’t show that the verifier is right. Some independent passes give partial evidence. A task passed once by an exploit is kept as well.
  5. A lab has plenty of organic English and no graded coding tasks. Where would you expect it to rely on generated data, and what could catch the generator’s mistakes in each place?

    AnswerTasks, where tests can catch a wrong patch, though they need checking themselves. If it rephrases English, only a model check against the source can catch a changed fact, and repeats or fresh lower-tier text are the alternatives. That’s where the trade-off in the callout is sharpest.

In the map

Methods

Taxonomy nodes

Further reading

↑