Open Models Almanac

Training stages · S2

Mid-training and long-context extension: one word for several operations

A stage on a finished base adds skills and context length, the model loses some of what it knew, and both sides are mostly measured through stand-ins

Illustration of a finished clay vase on a potter's wheel, a long coral thread being drawn from its neck across the table, and a few small shards and a little sand beside the wheel.

Nemotron 3.5 Super starts from an earlier base model and goes through six more stages before anyone fine-tunes it on conversations. The one NVIDIA calls mid-training reads 65.4B tokens at 256K tokens per sequence. Kolibri’s mid-training reads 3.44T tokens at 64K, 53 times as many, and changes different things. Eight of the 16 labs on this site use the word for a stage, and under it they do at least four kinds of operation.

These stages sit after broad pre-training on a finished base and before supervised fine-tuning. A base pre-trained on short sequences of mostly web text has seen few 200K-token repositories and few agents calling tools, because most documents are short and long sequences need more memory and compute. Running pre-training again with that data in it would cost about as much as the first run. A shorter stage on the finished base is the affordable route, and long-context extension, which lengthens the sequences, is its close relative.

Super shows what such a stage can add and what the model can lose. After the same short test fine-tune, the mid-trained model solved on average 8.6 percentage points more SWE-bench issues than the model before it, across five agent programs. Super had been extended to 1M tokens before this stage, and mid-training at 256K dropped its score on a 1M-token retrieval test from 79.9 to 33.5, until a 9B-token recovery stage at 1M brought it to 81.0. After a longer 100B-token test fine-tune, NVIDIA compared the two only under OpenHands, where the stage had shown no gain after the short one either. Mistral states the trade in one sentence: “Midtraining trades off acquiring new capabilities (e.g., agentic), against retaining the long tail of pretraining knowledge.”

Background

As in the primer, a run comes in stages. Pre-training is the long first stage on broad text. Supervised fine-tuning (SFT) trains on conversations and turns a base into an assistant, and reinforcement learning (RL) follows. The names in between overlap. Any stage that trains a finished base further on plain token prediction is, mechanically, continued pretraining (CPT). Labs call it mid-training when it shifts the data toward what post-training will need, long-context extension when it lengthens the sequences, and continued pretraining for almost anything else, including changing the architecture. This article sorts the stages by what they change, and gives each lab’s own word next to it.

A model trained only on 8K-token sequences hasn’t learned to use position 500,000. With rotary position embeddings (RoPE), far positions also produce rotations it never saw in training. T4 covers RoPE, rescalings such as YaRN, and NoPE (no position encoding in some layers). Some labs change the encoding when they extend, and others built models that don’t need it.

Token counts are tokens seen, counting repeats, unless the text says unique (01 has the kinds). Replay, which the continual-learning literature calls rehearsal, mixes some of an earlier stage’s data into a later one to slow forgetting, the loss of earlier skills. A learning-rate re-warm raises the rate again at the start of a stage after the previous stage decayed it, which the optimization literature calls a warm restart (SGDR, Loshchilov and Hutter, 2017). The optimizer’s running averages of past gradients, its state, can be carried into a new stage or reset. With the decay rates large-model runs typically use (0.9 and 0.95), Adam’s averages forget within a few dozen steps, so the choice matters mostly for whether a warm-up is needed.

An agent here is a model that solves a software task by calling tools, such as running a shell command or editing a file, and reading their output. The program around it, its harness or scaffold, names the tools, formats their results and prompts the model. OpenHands, Codex and Claude Code are harnesses, and a model can score very differently in two of them. SWE-bench Verified gives the agent a real GitHub issue and its repository, and counts the share of issues where its patch passes the tests, as the success rate of single attempts, averaged over several per issue (pass@1). SWE-bench Multilingual and Pro are harder variants, DeepSWE a harder set again, and Terminal-Bench a set of command-line tasks. RULER tests retrieval and tracing over inputs of a chosen length (RULER-1M at a million tokens). MATH-500 is competition math, TriviaQA recall of facts, and HellaSwag commonsense sentence completion. Labs often judge a base by a test SFT: the same small fine-tuning run applied to two bases, so the bases are compared after a step closer to what ships.

Nothing about these stages is in a model’s configuration or code, so we read technical reports first, then model cards and official blogs. We count labs, not models, and a blank means we searched and found nothing, which doesn’t mean the lab didn’t do it. Nemotron 3.5 Super is our worked example because its report and a companion paper describe its stages most fully. That makes it the most detailed account, not a typical one.

One word, several operations

We sort the stages called mid-training into four groups of our own: a data shift toward better data, agentic priming with software-engineering data and agent traces, a chat format stage that teaches the conversation format, and a bridge that also changes the optimizer or numeric precision on the way to RL. Several also lengthen the sequences. Several rows belong to more than one group.

Model (report)TokensWhat changesWhere the report puts itOur group
Kolibri3.44TData toward math, code, reasoning, instructions; 16k to 64kPre-trainingData shift
MiMo-V2-Flash (Xiaomi)4TCode upsampled, about 5% synthetic reasoningPre-trainingData shift
LFM2 (Liquid)1THigher-quality data; 4K to 32KPre-trainingData shift, length
Nemotron 3.5 Super65.4BNatural software-engineering data, 10% agent traces, at 256KPre-trainingAgentic
GLM-5 (Z.ai)1.55TIssue and pull-request data, agent trajectories; 32K to 200KPre-trainingAgentic, length
LFM33.5TMore agent traces and tool calls, chat data in the chat template; to 256KPre-trainingAgentic, chat format, length
MiMo-V2.6not statedAgent trajectories; new optimizer; 4-bit training; 256K, then 1MPre-trainingAgentic, bridge, length
DeepSeek V4not stated“agentic data”, in one sentencePre-trainingAgentic
Laguna XS.2 and M.1 (Poolside)about 60B, one epochChat format, new special tokens, reasoning, code and agentsPost-trainingChat format
Mistral Large 4not stated1/5 of the peak learning rate, agent trajectoriesPost-trainingBridge, agentic

Sizes run from about 60B to 4T tokens, a 61-fold span on tokens seen (65.4B to 4T, computed). The operations differ as well, even inside one lab. Xiaomi’s MiMo-V2-Flash report uses mid-training for a 4T-token data shift inside pre-training, while its MiMo-V2.6 report uses it for a stage after pre-training that adds agent data, the final step to 1M tokens, an optimizer switch and low-precision training. Liquid’s LFM2 report uses the word for three operations: a long-context stage for the language model, a vision fusion stage and an audio stage.

Six labs describe mid-training inside their pre-training section. Poolside and Mistral put it under post-training, and the taxonomy on this site follows them, filing the node under post-training. Moonshot extends Kimi K3’s context in four stages and never uses the word. Some of these reports describe models before the ones on this site: the site has Laguna S 2.1, whose blog names only SFT and RL, and GLM-5.2 and 5.3, which share the GLM-5 lineage.

Figure 1.The stages after broad pre-training, in each report's order, with each lab's own word for them. Blocks are equal-width and not to scale. Under Super's strip, its 1M-token retrieval score after three of the stages.

Data shift: better data on a finished base

The most common sense is a data shift. Pre-training’s mix is broad, and the best data is small, so a lab spends it late, on a base that already knows language (D2 covers how the mix is chosen). Kolibri’s mid-training is the largest stage of this kind with a published comparison. It gives 73.1% of 3.44T tokens to English instruction and reasoning data, STEM question answering and code, and 3.4% to web domains (computed from the report’s shares). The learning rate stays constant through all three base stages, and “Each stage (except pre-training) continues with the optimiser state after finishing the previous one and therefore needs no further warmup.” For the first 100B tokens the pre-training mix fades out linearly, a replay share that falls from all of the mix to none.

Aleph Alpha measured the stage on a proxy model with 0.65B active parameters (11.9B total), at 400B tokens, against a run that continued pre-training on the pre-training mix at 16K. The production mix led by 4.9 points on an average of base benchmarks, and by 11.5 after a test SFT. It also lost: against that control, TriviaQA fell 8.4 points and German HellaSwag 7.2. The control differs in sequence length too, and the TriviaQA drop appears in all three mixes tried, including one with 33% English web text, so Aleph Alpha writes that “the cause of the TriviaQA loss remains open.” The German drop has a clearer cause. On shorter 84B-token proxy runs, German HellaSwag rose from 65.7 with no German web text in the mix to 79.4 with 19%, so it tracks how much German text the mix keeps. That is rehearsal measured inside one lab: a skill holds better the more of its data stays.

The other data shifts are thinner on evidence. MiMo-V2-Flash’s 4T-token stage continues the pre-training cosine decay, from 1.0×10⁻⁴ to 3.0×10⁻⁵, so it has no boundary in the learning rate at all, only in the data. Liquid’s LFM2 adds “an additional 1T higher-quality tokens” with “an accelerated learning-rate decay schedule”. Neither compares the stage with an alternative.

Mistral is the one lab that reports calibrating the learning rate and the amount of replay together: “using 1/5 of the peak pretraining LR and 1/10 of pretraining rehearsal achieved the best compromise.” The report doesn’t say whether “1/10 of pretraining rehearsal” means a 10% replay share, and it prints no numbers for the compromise.

Agentic priming: the worked example

Why NVIDIA added the stage

“Pretraining runs at short context and sees almost no agent interactions, and the natural data around software engineering, such as whole repositories, pull requests with their issues, commit histories, and build logs, is often too long for it. SFT in turn consists mostly of synthetic data, and very long SFT stages can reduce how much the model later gains from RL.” The report doesn’t measure the second half. The companion paper, Singhal et al. (2026), supports it with citations and “we have observed this in our own training as well”, with no data. That paper is by NVIDIA’s Nemotron team, and the report cites it for the full recipe. It is the longest public account of an agentic mid-training stage, not an independent replication.

What goes in

Super’s 65.4B tokens (67B on the cards) at 256K are mostly natural software-engineering data: issue and pull-request pairs 40.8%, whole repositories 20.0%, commit diffs 3.0%, build and CI logs 1.2%. Another 20.0% is replay from the second phase of pre-training, and 5.0% is long-context data from the earlier extension. The last 10.0% is agent traces from many harnesses, in the Nemotron chat template; “Most of it is natural data kept in its original format, without chat templates.” On the traces the loss covers only the model’s own turns, and tool responses are masked out of it. Documents packed into one 256K sequence are masked from each other, and the multi-token prediction layers stay in the loss.

The paper’s ablations on Lightning research runs (30B total parameters, 100B tokens, full SFT) separate these parts. Natural data alone moved SWE-bench Verified from 59.7 to 60.5, and the whole blend to 63.5, so the traces account for 3.0 of the 3.8 points. Training on tool responses instead of masking them “lowered SWE-bench Verified pass@1 by about 10 points” in an earlier recipe. Replay at 40% gave “no clear benefit over 20%”, and no run without replay is reported. A higher learning rate favoured agentic gains and cost general skill. MATH-500 went from the base’s 79.7 to 78.8 at a peak rate of 1×10⁻⁵, 74.3 at 3×10⁻⁵ and 61.7 at 5×10⁻⁵ (single runs, with 20% replay), while early runs at 1×10⁻⁵ “gave smaller gains after SFT”. Lightning’s runs used 5×10⁻⁵ “which favors agentic gains”, and Super’s used 1×10⁻⁵, which “favors preserving general capabilities”.

Results after a test SFT

NVIDIA measured the stage with a short test SFT, “an experimental agentic blend of about 33B tokens (2K steps), not the one used in Section 3”, applied to the base before and after mid-training. Under five harnesses, SWE-bench Verified rose under every one, by 8.6 points on average, from 3.4 (Hermes) to 13.4 (Codex), and Terminal-Bench 2.1 by 6.8. DeepSWE, which NVIDIA calls its hardest benchmark, went from 6.4 to 10.4, averaged over four runs, against a run-to-run noise of about two points. Under OpenHands, evaluated separately, SWE-bench Verified moved from 71.3 to 71.9, inside a noise of “about one point”.

Figure 2.Super before and after its 65B-token mid-training. Top, SWE-bench Verified by harness after the same short test SFT. Bottom, four benchmarks after the full 100B-token SFT, the SWE-bench ones under OpenHands only. Grey bands are evaluation noise for one checkpoint pair, not variation between training runs. None of these is the released model's score.

Part of this gain may be familiarity with the harnesses. The stage’s traces come from more than ten of them, and in an early snapshot Codex supplied 19.4% of trajectories and OpenHands 14.0%, against 5.2% for OpenCode and 1.3% for Claude Code. Codex had the largest gain and OpenHands none, so trace share doesn’t predict the gain either, and the paper says its results “do not separate the effect of scaffold diversity from that of adding traces.” The six rows are one checkpoint pair after one test SFT each, so they show that these two checkpoints differ, not how much a repeated run would.

Results after the full SFT

Singhal et al. also print Super after the full 100B-token SFT, under OpenHands only, the harness where the short SFT had already shown no gain. There the 65B stage, the size that shipped, is level with or below the base on every SWE-bench variant: Verified 72.9 against 72.5, Multilingual 69.0 against 68.2, Pro 53.9 against 52.4. DeepSWE rises from 11.7 to 13.5, inside its two-point noise. A longer 280B-token research run raised Verified to 74.3 and DeepSWE to 15.7 after the full SFT, and left Multilingual and Pro 1.1 and 2.1 points below the base, which the authors “do not read … as a consistent effect”. The largest gains are on Lightning research runs of up to 500B tokens, with SWE-bench Verified from 60.7 to 64.5 and Multilingual from 55.8 to 62.8, but the released Lightning has no mid-training stage. A 100B-token Lightning research run is the one model compared across harnesses after the full SFT, and there the gain held under all six, from 3.8 points under OpenHands to 19.0 under Codex. Base-model scores rose with mid-training “but did not track post-training results closely enough to guide recipe decisions”.

NVIDIA’s stated case for the stage is that it lets SFT stay short and leaves RL more room, and the RL evidence for it is thin. On a Lightning research run, 20 steps of RL with one seed kept the mid-trained model’s two-point lead, with fewer format violations (0.03% of rollouts against 0.34%) and trajectories about 15% faster. The authors write that this does “not show that midtraining increases what RL can add”.

Six more labs put agent data in a mid-training stage, none with a controlled comparison: Z.ai (GLM-5, about 160B unique tokens of issue and pull-request data among its 1.55T tokens), Xiaomi (MiMo-V2.6), Liquid (LFM3), Poolside (30% “coding-and-agent” data), Mistral, which keeps agent trajectories and software tasks “whose tests cannot be validated” for midtraining, and DeepSeek, in one sentence of its V4 report.

Chat format, and the stage as a bridge

Poolside’s mid-training for Laguna XS.2 and M.1 runs one epoch over about 60B tokens at up to 131,072 per sequence: 40% logic and reasoning, 30% code and agents, 30% general chat. It introduces “new XML-like special tokens to be used to format multi-turn chats”, and the SFT that follows “shares the same batch size, sequence length, packing, learning-rate schedule and optimizer as mid-training”. The two stages therefore differ mainly in data and in epochs (one against three), and Poolside’s mid-training is close to a first SFT. LFM3’s mid-training also includes “chat data with and without thinking traces, formatted with the same chat template as LFM3-1.7B.” NVIDIA’s keeps all but its 10% of traces out of any template.

New tokens need new embedding rows. On M.1, “the mismatch between special and regular token embeddings caused gradient spikes, dead experts, and broader training instabilities”. Poolside fixed it by initializing the new rows from the average of their subword pieces and warming up for 100 steps with the rest of the model frozen.

A stage between pre-training and RL is also a convenient place to change other things. MiMo-V2.6 switches its hidden weight matrices from AdamW to Muown, a variant of Muon, “to prepare the model for subsequent large-batch RL” (O1), starts training in the 4-bit MXFP4 format it will be served in (O2), and adds examples built from reward hacking seen in RL (S4). The only measurement is that “we observe no loss spike”. MiMo-V2-Flash lowers the weight of its multi-token prediction loss from 0.3 to 0.1 at mid-training. Nothing in the reports separates the effects of these changes from those of the stage’s new data.

Long-context extension

Why it’s a separate stage

Ten labs describe training at a longer sequence length after pre-training starts: NVIDIA, Aleph Alpha, Poolside, Xiaomi, Liquid, Moonshot, Z.ai, DeepSeek, Qwen, and Mistral, which goes from 4K to 40K in the second phase of pre-training without saying how it reaches its stated 1M. MiniMax describes extension only in an experiment. Trained lengths run from 32K (LFM2) to 1M. Offered lengths can be longer: Kolibri trains to 256K and is offered up to 1M, Laguna goes from 128K to 256K “without any training by doubling the RoPE scale in the global attention layers”, Qwen’s cards reach about 1M with YaRN at inference, and this site lists GLM-5.2 at 1M where the GLM-5 report trains to 200K.

Extending needs an encoding that doesn’t break at new positions, and some training at the new length. Xiong et al. (2023) found Llama 2 “unable to effectively attend beyond 4,000 - 6,000 tokens even after extensive long-context continual pretraining” until they raised RoPE’s base frequency from 10,000 to 500,000. Fu et al. (2024) showed the training is cheap: 500M to 5B tokens of continued training at 80K gave a 7B model retrieval across 128K.

Pre-training stays short for three reasons. Most documents are short (in Fu et al.’s data, about 30% exceed 4K tokens). A long sequence doesn’t fit on one accelerator, so it is split across several with context parallelism: 32-way for Nemotron 3 Ultra at 1M, 16-way for Super and 8-way for Lightning. And for the attention layers, each token costs more the longer the sequence.

In training, the matrix multiplications of the dense layers cost about 6N6N floating-point operations (FLOPs) per token for NN active parameters: two per multiply-add, times three for the forward and backward passes. A full-attention layer with hh query heads of width dhd_h adds, at sequence length TT, a dot product with each earlier key and a weighted sum over the values, for T/2T/2 earlier tokens on average. That is 2⋅2⋅(T/2)⋅hdh2 \cdot 2 \cdot (T/2) \cdot h d_h forward, and 6Thdh6 T h d_h with the backward pass. Super has 8 attention layers of 32 heads of width 128 among 88 blocks, the rest being Mamba state-space layers and expert layers, whose cost doesn’t grow with TT. Kolibri has 10 full-attention layers of 48 heads of width 128, and 40 layers that attend only to the 512 preceding tokens (configs layers_block_type, layer_types, sliding_window: 513).

By this count, a training token at 1M costs Super about 3.8 times one at its 8K pre-training length, and a token at 256K costs Kolibri about 4.2 times one at its 16K (computed). They are rough: the attention in Super’s two multi-token prediction layers, which stay in training, would raise its 1M figure to about 4.5, and document masking, below, lowers both. They count the half of the attention matrix that causal kernels compute, and counting the full matrix, as Xiong et al. and most scaling papers do, gives 6.4 and 6.3. More important, labs mask attention between documents packed into one sequence, so a token pays for the length of its own document. The TT to use is then the length-weighted mean document length, E[L2]/E[L]\mathbb{E}[L^2]/\mathbb{E}[L] for document lengths LL, which is at most TT. If Super’s mid-training documents averaged 32K in that sense, a token would cost 1.07 times an 8K one, not 1.68. In the recipe Kolibri chose for its long-context stage, about 13% of tokens are in documents over 128K (measured on its ablation mix).

dots: lengths each model was trained atdotted line: 1×

Nemotron 3.5 Super at 256K: 1.68× a token at 8K, causal count (half the attention matrix).

Dense layers 58%, full attention 42%, no sliding-window layers.

Figure 3.Approximate training cost per token by length, against each model's pre-training length, counting only matrix multiplications (computed from each config). With documents masked from each other, read the axis as document length: tokens from short documents cost little even inside a 1M sequence.

Moonshot gives this as its reason for staging Kimi K3’s extension: “Concentrating the costly long-sequence computation within a small fraction of the overall training budget keeps the curriculum economical”. Packing brings its own systems cost: when documents of uneven length share a step, “every rank waits for the rank with the most expensive sequence”, and Aleph Alpha raised throughput by 84% in a 256K fine-tuning run by sorting sequences by attention cost. Moonshot also built context parallelism for its linear-attention layers (KDA), and Aleph Alpha credits its sliding-window hybrid with allowing “longer training sequences than in Kolibri Origin” in pre-training itself.

Staged or direct

Labs disagree on how many steps to take, and no source compares the two at scale.

Figure 4.The sequence lengths each base was trained at, from pre-training to the last extension, on a log scale. Arrows give the step ratio, and tokens are printed where the report gives them. Super's dashed return is its 256K mid-training and the recovery that followed.

NVIDIA’s Nemotron 3 report states, of its Nano model: “In CPT, we did not observe the need to follow a staged increase of training sequence length from 8k to 512k.” Its Nemotron 3.5 models also go from 8K to 1M in one stage. NVIDIA’s hybrids use no RoPE, so they “do not suffer from out-of-distribution RoPE issues during context extension”. Moonshot stages for cost, and its model uses no RoPE in attention either: it “extrapolates directly to 1M-token contexts without any positional-encoding modification, such as RoPE rescaling or interpolation”. Since both avoid the RoPE problem and still choose differently, the position encoding doesn’t explain the split. Z.ai says its last 200K stage “further bolstered the model’s performance even within the 128K context window”, with no numbers. The nearest comparisons are on smaller models and about when to switch. Xiong et al. found that training at 4K for the first 80% of tokens and at 32K after saved about 40% of FLOPs against 32K throughout, with little loss on most long-context question-answering tasks. Press et al. (2021) found short-then-long training faster than long throughout on small language models, a form of curriculum learning.

Where it goes, and what happens to position

Placement varies even inside one lab. Xiaomi extends after mid-training in MiMo-V2-Flash, partly in pre-training and partly at the end of mid-training in MiMo-V2.6, and in “SFT & Agentic Post Training” according to the MiMo-V2.5 cards. DeepSeek V4.1 extends from 64K to 1M at 34T of 45T tokens, during its learning-rate decay.

Position is handled in four ways (T4). NVIDIA uses no RoPE, Kimi K3 none in its attention layers, and Kolibri none in its full-attention layers, keeping RoPE only in sliding-window layers whose window never grows; all three extend without changing anything. Laguna applies YaRN “to global attention layers only”. MiMo-V2-Flash raises the RoPE base of its global layers from 640,000 to 5,000,000. Qwen’s cards reach about 1M at inference with a fixed YaRN factor, and warn that this “potentially impact[s] performance on shorter texts”.

Re-warm the learning rate or not

At the extension boundary, Laguna doesn’t re-warm: “in our experiments this transferred better than reintroducing a warmup at the start of context extension.” Kolibri carries on with the same constant rate and optimizer state. NVIDIA holds its extensions at a constant low rate (4.5×10⁻⁶ for Super), and MiMo-V2-Flash and GLM-5 continue the decay already running. None prints the comparison. A constant rate isn’t a constant step either: Kolibri’s batch grows from 75.5M tokens per step in pre-training to 100.7M in mid-training and 201.3M in the long-context stage, Ultra holds tokens per iteration fixed, and MiMo-V2-Flash cuts its batch from 2,048 sequences to 256 as the length grows eightfold, so its tokens per step stay at about 67M.

Outside papers compare re-warming and agree on the trade. Re-warming speeds learning of new data and raises loss on old data. Ibrahim et al. (2024) found that re-warming and re-decaying “improves adaptation to a new dataset”, and that re-warming even on unchanged data is “a significant cause of the increase in loss”. Parmar et al. (2024, NVIDIA), polishing a 15B model on mostly old data, found that “any level of warmup” lowered accuracy, by at most 0.9 points in single runs. Ibrahim’s goal was adaptation and Parmar’s retention, so they chose differently. Neither tested a long-context stage. NVIDIA’s own recipe does each in a different stage: it re-warms for mid-training, to about twice its final pre-training rate, and holds the extension at a constant low rate.

Keeping short context, and the benchmark question

Labs report that training only on long sequences costs short-context skill, and they keep short data in the mix. Nemotron 3 Ultra’s extension spends 8% of its iterations at 4K “to maintain the accuracy of the short benchmarks”. Kolibri takes one third of its extension tokens from naturally long documents, one third from mid-training data reweighted toward length, and one third from mid-training data as it was, because training “on long documents alone … tends to erode the skills acquired earlier”. Kimi K3 upsamples long documents “so that the long-context distribution is not overwhelmed by short sequences during cooldown”. MiMo-V2-Flash keeps its mid-training mix and upsamples data with long-range dependencies.

Labs disagree on synthetic long-context data. Aleph Alpha removed synthetic samples from its long pool “to avoid artificially favourable results on synthetic long-context benchmarks such as RULER”, and for Ultra NVIDIA writes “We did not use any RULER-style data in the blend.” Others train on such data on purpose. Z.ai adds “a small proportion of MRCR-like data” at 200K (MRCR asks a model to find one of several similar requests in a long conversation). Moonshot synthesizes long-range tasks from permuted and concatenated documents, because “Length alone, however, does not confer long-range capability.” NVIDIA’s own Nemotron 3 Nano extension “included synthetic data designed to support long-range retrieval”. The outside papers split the same way: Fu et al. found natural long documents gave poor retrieval until upsampled, and Xiong et al. that “having abundant long texts in the pretrain dataset is not the key”. Their tests differ (retrieval at 128K, question answering at 16K), and no source compares the two policies on one model.

Benefit can be hard to measure. Kolibri’s ablation varied only the data at a fixed 256K length, and a control trained on mid-training documents stayed close overall, with the long-document mix ahead mostly on retrieval at the longest lengths (01). Moonshot reports no long-context benchmark and no token counts for Kimi K3’s stages.

Forgetting, and getting it back

Super was extended to 1M first, then mid-trained at 256K, and RULER-1M fell from 79.9 to 33.5. A 9B-token “long-context recovery phase” at 1M, “with the same recipe” as the extension (on long-context data only, says the companion paper, which prints 33.0 after 25B tokens of mid-training), brought it to 81.0, above where it started. The paper calls the drop “a side effect of our evaluation setup”, since mid-training started from the 1M checkpoint, and recommends that “In a mature pipeline, midtraining would come before the final long-context stage”.

It tests putting mid-training before the extension on Lightning research runs, starting from the 8K base, against the extension alone.

Figure 5.RULER on Nemotron 3.5 Lightning research runs, by evaluation length (Singhal et al., Table 8). The two-stage path reads 100B more tokens and different data, and no run tries the stages in the other order. None of these is the released Lightning.

No run in the paper tries Super’s shipped order the other way round, so this shows that adding the stage before the extension helps, not which order is better. Super’s drop is also a model evaluated at four times the length it had last trained at, which a few billion tokens back at 1M restored. That fits a skill that stayed recoverable rather than one erased, but it doesn’t show it. The continual-learning literature calls the test for this savings: relearning should take fewer examples than learning from scratch. The recovery took 9B tokens against the original extension’s 35B, but it started from a model trained at 256K, a 4-fold step from 1M where the original step was 128-fold, and Fu et al. show such steps are cheap anyway. Telling the readings apart needs the same 9B tokens applied to a 256K model that never saw 1M, and nobody reports that run.

Replay keeps what it samples. With 20% replay, Super’s knowledge, math, code and commonsense groups stayed “within 1.2 points of the starting checkpoint” through 65B tokens. The replay was packed at 256K, so it held nothing at the length where RULER-1M was lost, and it didn’t protect that score. Outside studies measured how much replay a shift needs. Ibrahim et al. found that “even the lowest tested replay of 1% significantly reduces forgetting” and chose 5% for a weak shift and 25% for a strong one, while 50% slowed learning of the new data. Their models are up to 10B parameters trained on 300B tokens, far less than today’s bases. No lab reports a run without replay.

A drop before and after a stage is also not yet its cost, which is the difference from a run that did something else with the same tokens. Aleph Alpha has that comparison only on its proxy (above). NVIDIA writes that “We did not compare against spending the same tokens on longer pretraining or SFT”.

Continued pretraining: three uses of one name

Labs write “continued”, “continual” and “continuous” pretraining. NVIDIA’s documents use the phrase for four stages: the Indic top-up, the long-context extension, a warm-up of the multi-token prediction head with the backbone frozen, and a diagnostic branch in BF16. The recovery reuses the extension’s recipe. The uses fall into three kinds: a long-context or quality top-up, as above (Qwen3.5-Omni’s speech module gets one too), an architecture change, and a domain top-up.

Architecture adaptation

Here the model changes more than the data. A lab swaps dense attention for a sparse kind on a trained checkpoint, where each query attends only to the keys a small indexer selects (T3). Four labs describe this for a model they shipped, and the recipes agree in shape. First, freeze the model and train only the new indexer to match the dense attention it will replace, with a Kullback-Leibler (KL) loss toward the attention scores summed over heads, for about 1,000 steps where the step count is given. Choosing the top kk keys is a discrete step, so the language-model loss passes no gradient to the indexer, and it learns from this distillation loss instead. The warm-up comes first so that when sparsity starts, the indexer already picks keys the dense model attended to. Second, train under sparse attention. DeepSeek keeps the indexer on its distillation loss then, now over the selected keys only, and keeps the language-model loss away from it.

The KL loss bounds what selection misses. If the indexer’s distribution over keys is within KL divergence DD of the head-summed attention, the attention mass its top kk leave out, beyond what the best kk would, is at most 2D\sqrt{2D} (Pinsker’s inequality). That holds for the head sum, not for each head.

Figure 6.Tokens spent converting a trained model to sparse attention, in two steps: training the new indexer alone, then training under sparse attention. Computed from each report's steps × sequences × length where given. Rows differ in model size and sequence length, so the spread is in tokens, not cost. MiniMax's row is an experiment; its reports don't say how the shipped M3 got its sparse attention.

DeepSeek V3.2 converts V3.1-Terminus with 2.1B tokens of warm-up and 943.7B of sparse training at 128K, on “the 128K long context extension data used for DeepSeek-V3.1-Terminus”. GLM-5’s step is headed “Continued Pre-Training with DeepSeek Sparse Attention (DSA)”, starts “from the base model at the end of mid-training”, and uses that stage’s data for 2.84B tokens of warm-up and 20B of sparse training. Qwen3.8-Flash-Next introduces its sparse attention “during the continued pretraining (CPT) stage … with a sequence length of 256K tokens”, for about 2B and 200B. Mistral warms up for about 15.1B tokens and trains for 198B, and doesn’t say where that sits among its stages.

Each claims to lose little, at its own scope. DeepSeek compared V3.2-Exp with V3.1-Terminus and does “not observe substantial performance degradation … on both short- and long-context tasks”. Z.ai’s sparse and dense models “tie in training loss and evaluation benchmarks” after the same SFT, and before SFT, at 128K, the sparse model ties on one retrieval task, leads on two and trails by 3.3 points on HotpotQA (63.0 against 66.3). Qwen’s sparse model beats its full-attention version on RULER from 512K to 1M (93.00 against 90.08), beyond the 256K it was trained at, though on MRCR with eight needles at 1M both score under 27. On a smaller test model, GLM-4.7-Flash, the warm-up alone dropped RULER at 128K from 79.21 to 71.35, and 150B tokens of sparse training, 7.5 times what GLM-5 spent, brought it back to 78.86. MiniMax’s 109B experiment loses 2.6 points on RULER at 8K against full attention (and gains 0.7 at 32K), is level at 128K after a further 140B tokens of extension, and loses 7.0 against the same architecture trained sparse from scratch on the same 3T-token budget.

DeepSeek has taken three routes in three reports: conversion by CPT (V3.2), a 1T-token dense warm-up inside pre-training before sparse attention starts (V4), and sparse attention from scratch at 64K “without any dense attention warmup stages” (V4.1). Mistral made a different change by another route. To remove a form of KV-cache compression from an early design, it modified “the architecture of an existing checkpoint prior to the second phase of pretraining”, so the rest of pre-training was also its recovery.

Domain top-up

Super’s Indic CPT has a “nominal 100B-token budget, with 10% (10B tokens)” of Indic data, translated and natural, and the other 90B “retains the preceding training mixture’s relative source proportions”. These are sampling allocations, “not measured per-source token totals”. It raised five Indic benchmarks by 4.98 to 9.82 points against the checkpoint before it, and NVIDIA “observed no regressions in the tested categories” in English, with no table. Lightning research runs found that higher Indic shares helped further, “but with smaller incremental gains”.

Comparison

OperationWhat changesSizesBest measured effect, with baselineMeasured cost
Data-shift mid-trainingMix toward reasoning, code, instructions1T to 4T tokensKolibri proxy (0.65B active) at 400B tokens: +11.5 after a test SFT against continued pre-training, which also differs in lengthSame comparison: TriviaQA −8.4, German HellaSwag −7.2
Agentic mid-trainingNatural software data, agent traces, long sequences65.4B to 3.5T tokens, or unstatedSuper after a 33B-token test SFT: +8.6 SWE-bench Verified under five harnesses, +0.6 under OpenHands; after the full SFT, −0.4 under OpenHandsRULER-1M 79.9 to 33.5; MATH-500 61.7 against 78.8 at a 5× higher rate (Lightning research run)
Chat-format mid-trainingChat template, special tokens, reasoning, toolsabout 60B tokens (Poolside)none publishedinstability from new token embeddings (M.1)
Long-context extensionSequence length, sometimes position encoding and data21B to 1T tokens, or unstatedKolibri: retrieval at 256K over a same-length control; changes after SFT within noiseabout 4× per token at the trained length counting the causal half of attention, about 6× counting the full matrix (two hybrids, computed); context parallelism
Recovery stageSame recipe as the extension9B tokensSuper RULER-1M 33.5 to 81.0one more stage
Architecture CPTAttention type, with a new indexer23B to 946B tokens“tie” after SFT (GLM-5); “not … substantial performance degradation” (DeepSeek V3.2-Exp)MiniMax experiment: RULER-8K −2.6 against full attention, −7.0 against sparse from scratch
Domain CPT10% new-language data100B tokens nominalSuper: +4.98 to +9.82 on Indic benchmarks“no regressions” in English, no table

Where the labs sit

One row per lab and report, in each report’s terms, with tokens seen unless marked. A blank means we found nothing. Where the report describes a model before the one on this site, the row says so.

LabModel (report)Stage word(s)TokensTrained length (offered)Learning rate at the boundary
NVIDIANemotron 3.5 SuperMTP warm-up; distillation; Indic continued pretraining; LC-Phase; mid-training; long-context recovery100B; 1T; 100B; 35B; 65.4B (67B on cards); 9B1MMid-training reset and warmed to 1×10⁻⁵, about twice the final pre-training rate; extension constant 4.5×10⁻⁶
NVIDIANemotron 3.5 LightningLC-Phase only (“did not have a mid-training stage”)21B1Mconstant 1×10⁻⁵
NVIDIANemotron 3 Ultracontinuous pretraining (LC-Phase)33B, 8% of iterations at 4K1Mconstant 2.5×10⁻⁶
Aleph AlphaKolibrimid-training; long-context extension3.44T (3.4T in the overview); 200B (201B on the card)256K (1M)constant throughout, optimizer state carried; batch 75.5M, 100.7M, 201.3M tokens per step
Mistral AIMistral Large 4midtraining (under post-training); sparse-attention adaptationnot stated; 15.1B + 198B40K in pre-training (1M)1/5 of peak pre-training rate
PoolsideLaguna XS.2 and M.1 (report; Laguna S 2.1 on this site describes neither stage)long-context training; mid-training (under post-training)100B + 100B; about 60B, one epoch128K (256K, no training)extension not re-warmed; mid-training re-warmed to 1×10⁻⁵
XiaomiMiMo-V2-Flash (report)Stage 2 mid-training; Stage 3 context extension4T; 1T256Kone cosine decay, continuous
XiaomiMiMo-V2.6mid-trainingnot stated1Mnot stated
Liquid AILFM2 (report); LFM3long-context mid-training; mid-training1T; 3.5T32K; 256Kaccelerated decay (LFM2)
Z.aiGLM-5 (report; GLM-5.2 and 5.3 on this site share its lineage)Mid-Training; Continued Pre-Training with DSA1.55T (1T + 500B + 50B); 2.84B + 20B200K (GLM-5.2 listed at 1M)linear decay 4×10⁻⁵ to 1×10⁻⁵, continuous; DSA warm-up at up to 5×10⁻³, then constant 1×10⁻⁵
Moonshot AIKimi K3“a subsequent training phase”; cooldownnot stated1Mone cosine schedule
DeepSeekDeepSeek V4; V4.1V4: “mid-training phase” inside pre-training; V4.1: nonenot stated1Mone schedule for the run
DeepSeekDeepSeek V3.2continued pre-training (sparse attention)2.1B + 943.7B128K10⁻³ indexer, then 7.3×10⁻⁶
QwenQwen3.8-Flash-Nextcontinued pretraining (sparse attention)about 2B + 200B256K (about 1M with YaRN)10⁻³ indexer, then 2.5×10⁻⁵
MiniMaxMiniMax M3sparse continued pretraining, long-context training (109B experiment only)40B + 360B; about 140B(1M)not stated

Google DeepMind, OpenAI, StepFun, Tencent and Thinking Machines describe no stage between pre-training and post-training in the documents we have. Gemma 4 follows “a similar pre-training as Gemma 3”, and gpt-oss extends its dense layers to 131,072 tokens with YaRN without describing a stage.

Easy to get wrong

  • “Mid-training” in two reports may be two operations. Before comparing sizes, check what changed: data, length, format or optimizer. Kolibri’s 3.44T tokens and Super’s 65.4B don’t make Kolibri “more mid-trained”.
  • Super’s harness gain was measured after a 33B-token test SFT. After the full SFT, measured under OpenHands only, the 65B stage that shipped shows no SWE-bench gain, and the clear full-SFT gains come from 280B- and 500B-token research runs and from a Lightning research run under other harnesses. Singhal et al. is NVIDIA’s account of NVIDIA’s stage.
  • “We did not observe the need to follow a staged increase” is from the Nemotron 3 report, about extending its Nano model to 512K. Nobody has published staged and direct extension on one model.
  • A model’s offered context isn’t its trained length. Kolibri trains to 256K and is offered to 1M, Laguna doubles its length without training, and Qwen reaches 1M with YaRN at inference. A good RULER score beyond the trained length can sit beside a poor MRCR score at the same length.
  • The sparse-attention parity claims hold at their scope: V3.2-Exp against V3.1-Terminus, GLM-5 after one SFT, Qwen against its own full-attention CPT. MiniMax’s conversion is a 109B experiment, and its shipped route isn’t stated.

Check yourself

1. Nemotron 3 Ultra’s extension holds 25,165,824 tokens per iteration, 1M-token and 4K-token sequences are never mixed within one iteration, and 8% of iterations are at 4K. How many sequences is a 4K iteration, and about how many of the stage’s 33B tokens are short?

25,165,824 / 4,096 = 6,144 sequences (a long iteration is 24 sequences of 1,048,576). Since every iteration has the same number of tokens, 8% of iterations is 8% of tokens: 0.08 × 33B ≈ 2.6B tokens at 4K.

2. Using the article’s cost formula, why does a 1M token cost Super about 3.8 times an 8K token, when Super has only 8 attention layers? When would the real ratio be lower?

The dense part is about 6 × 12B = 7.2×10¹⁰ FLOPs per token at any length. Each attention layer adds 6 × T × 32 × 128, which at T = 1,048,576 is about 2.6×10¹⁰, and 8 layers give about 2.1×10¹¹. The total, about 2.8×10¹¹, is about 3.8 times the 7.4×10¹⁰ at 8K. Few attention layers delay the point where attention matches the dense layers (about 358K), but don’t remove it. The real ratio is lower when the 1M sequence is packed from shorter documents masked from each other, since a token then pays for its own document’s length.

3. Super’s mid-trained model scores 8.6 points higher on SWE-bench Verified under five harnesses after a short test SFT. A colleague says the stage was worth it. What would you check before agreeing?

Which fine-tune came after, and in which harness: after the paper’s 100B-token SFT, measured under OpenHands only, the 65B stage is level with or below the base on SWE-bench, and only longer research runs keep a gain there. Whether the harnesses were in the training data: the stage’s traces come from many harnesses, Codex the largest. How much the numbers vary: these are one checkpoint pair, with evaluation noise of about one point and no repeated training runs. And what the stage was for: NVIDIA’s stated aim is a shorter SFT that leaves RL more room, and that has no measurement yet.

4. Kolibri’s mid-training lowered TriviaQA by 6.8 points from pre-training to the end of mid-training. Why isn’t that, by itself, the cost of mid-training, and what does the closest thing to a cost show?

A cost is a difference from what the same tokens would have bought otherwise. Aleph Alpha has that comparison on its proxy at 400B tokens, where the production mix trails continued pre-training by 8.4 points on TriviaQA while leading by 11.5 after a test SFT. Even there the cause is open: the drop appears in mixes with a lot of English web text too, and the control trains at a shorter length.

In the map

Methods

Taxonomy nodes

Further reading

↑