Open Models Almanac
Methodspost-trainingmid-training & continual pretraining

specific method · filed under post-training

Long-context recovery training

A follow-up long-context training stage used to recover capabilities or evaluation performance after mid-training.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See mid-training & continual pretraining for the methods filed beside it.

Also called recovery stage, Long-context recovery, Long-context recovery training stage, long-context recovery stage.

sources
4
models
2
lab adopt it
1
strongest
used

Documented in

Evidence

6 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
used 6

we added a short long-context recovery phase after mid-training, for which we trained 9B tokens with the same recipe and successfully recovered the RULER score to 81.0.

usedtraining objectivein Nemotron 3.5 SuperNVIDIA
5 more from this model

To recover long-context capabilities affected by 256K mid-training, we perform an additional 9 billion token recovery stage

usedtraining objectivein Nemotron 3.5 SuperNVIDIA

a final 9-billion-token recovery stage at 1-million-token context restores the score from 33.5 to 81.0

usedtraining objectivein Nemotron 3.5 SuperNVIDIA

we followed mid-training with a short long-context recovery stage of about 9B tokens

usedpost trainingin Nemotron 3.5 SuperNVIDIA

a final 9B-token long-context recovery stage

usedtraining objectivein Nemotron 3.5 Super VLNVIDIA

a final 9B-token long-context recovery stage

usedtraining objectivein Nemotron 3.5 Super VLNVIDIA

we added a short long-context recovery phase after mid-training, for which we trained 9B tokens with the same recipe and successfully recovered the RULER score to 81.0.

usedtraining objectivein Nemotron 3.5 SuperNVIDIA
2 more from this model

To recover long-context capabilities affected by 256K mid-training, we perform an additional 9 billion token recovery stage

usedtraining objectivein Nemotron 3.5 SuperNVIDIA

we followed mid-training with a short long-context recovery stage of about 9B tokens

usedpost trainingin Nemotron 3.5 SuperNVIDIA
↑