we added a short long-context recovery phase after mid-training, for which we trained 9B tokens with the same recipe and successfully recovered the RULER score to 81.0.
5 more from this model
To recover long-context capabilities affected by 256K mid-training, we perform an additional 9 billion token recovery stage
a final 9-billion-token recovery stage at 1-million-token context restores the score from 33.5 to 81.0
we followed mid-training with a short long-context recovery stage of about 9B tokens
a final 9B-token long-context recovery stage
a final 9B-token long-context recovery stage