Open Models Almanac
Methodspost-trainingmid-training & continual pretraining

general family · filed under post-training

Mid-training

Mid-training is a stage between pretraining and SFT that trains the base model at long context on natural software-engineering data with a small share of agent traces.1

Trade-off Mid-training at 256K context hurt long-context performance, requiring a separate recovery stage to restore the RULER score.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Mid-training sits between the two. It trains the base model at long context on a diverse mixture of natural software-engineering data with a small share of agent traces — NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  2. after mid-training which was performed at a context length of 256K, we observed that the RULER-1M score dropped from 79.9 to 33.5. So we added a short long-context recovery phase — NVIDIA Nemotron 3.5 Super and Lightning Technical Report

Also called Mid-training phase.

sources
5
models
5
labs adopt it
4
strongest
core

Documented in

Evidence

6 quotes from 5 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1used 3core 2

Mid-training sits between the two. It trains the base model at long context on a diverse mixture of natural software-engineering data with a small share of agent traces, and leaves behavior shaping to post-training.

coretraining objectivein Nemotron 3.5 SuperNVIDIA
1 more from these models

To measure the effect of mid-training, we ran identical SFT on the base checkpoint and on the mid-trained checkpoint and compared the two.

evaluatedpost trainingin Nemotron 3.5 SuperNVIDIA
↑