Open Models Almanac
Methodspost-trainingmid-training & continual pretraining

general family · filed under post-training

Long-context extension via continued pretraining

Long-context extension via continued pretraining is a training stage that extends a model's context window after pre-training and mid-training.1 It continues training the existing checkpoint on longer sequences, raising the context length step by step across the stages.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. The long-context extension then adds 200B training tokens at 256k token context length — tech report
  2. Mid-training starts from the pre-training checkpoint at step 264 750 and long-context extension from the mid-training checkpoint at step 34 200 — tech report

Also called 21-billion-token long-context phase, extends the context window to 256K tokens, Long-context continued pretraining, Long-context training, long-context extension, three stages.

sources
12
models
6
labs adopt it
4
strongest
core

Documented in

Evidence

16 quotes from 12 sources, one card per model, each opening with its strongest.

How sources treat it
used 15core 1

In the LC-Phase, we performed continued pretraining (CPT) at 1M-token context length to equip the base model with long-context ability.

coretraining objectivein Nemotron 3.5 Super and LightningNVIDIA
6 more from this model

Lightning undergoes a 21-billion-token long-context phase that extends it to a 1-million-token context length

usedtraining objectivein Nemotron 3.5 LightningNVIDIA

A 35-billion-token continued-pretraining stage extends Super to a 1-million-token context length.

usedtraining objectivein Nemotron 3.5 SuperNVIDIA

Both models are trained for context lengths up to 1 million tokens.

usedtraining objectivein NVIDIA Nemotron 3.5 Super and NVIDIA Nemotron 3.5 LightningNVIDIA

Following distillation, both models undergo long-context training.

usedtraining objectivein Nemotron 3.5 Super and Nemotron 3.5 LightningNVIDIA

a 35B-token extension to 1M context

usedtraining objectivein Nemotron 3.5 Super VLNVIDIA

a 35B-token extension to 1M context

usedtraining objectivein Nemotron 3.5 Super VLNVIDIA

In the LC-Phase, we performed continued pretraining (CPT) at 1M-token context length to equip the base model with long-context ability.

coretraining objectivein Nemotron 3.5 Super and LightningNVIDIA
2 more from this model

we included a continued pre-training (CPT) stage at a 512k sequence length

usedotherin Nemotron 3 NanoNVIDIA

Following distillation, both models undergo long-context training.

usedtraining objectivein Nemotron 3.5 Super and Nemotron 3.5 LightningNVIDIA

LFM3-1.7B

Liquid AIadopted3 sources

This stage also extends the context window to 256K tokens.

usedmodel architecturein LFM3-1.7B-BaseLiquid AI
2 more from this model

This stage also extends the context window to 256K tokens.

usedmodel architecturein LFM3-1.7B-BaseLiquid AI

This stage also extends the context window to 256K tokens.

usedmodel architecturein LFM3-1.7B-BaseLiquid AI

Kolibri-1

Aleph Alphaadopted2 sources

We trained Kolibri on 768 B200 GPUs in three stages: 20T tokens of pre-training at a 16k sequence length over 21 days, 3.44T tokens of mid-training at 64k, and 200B tokens of long-context adaptation at 256k.

usedtraining objectivein KolibriAleph Alpha
2 more from this model

by interleaving naturally long documents with high-quality mid-training data rather than training only on long-context data

usedunclearin KolibriAleph Alpha

we interleaved long documents with the high-quality mid-training data of the previous stage.

useddata curationin KolibriAleph Alpha
↑