general family · filed under post-training
Mid-training
Mid-training is a stage between pretraining and SFT that trains the base model at long context on natural software-engineering data with a small share of agent traces.1
Trade-off Mid-training at 256K context hurt long-context performance, requiring a separate recovery stage to restore the RULER score.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Mid-training sits between the two. It trains the base model at long context on a diverse mixture of natural software-engineering data with a small share of agent traces
— NVIDIA Nemotron 3.5 Super and Lightning Technical Reportafter mid-training which was performed at a context length of 256K, we observed that the RULER-1M score dropped from 79.9 to 33.5. So we added a short long-context recovery phase
— NVIDIA Nemotron 3.5 Super and Lightning Technical Report
Also called Mid-training phase.