In the LC-Phase, we performed continued pretraining (CPT) at 1M-token context length to equip the base model with long-context ability.
6 more from this model
Lightning undergoes a 21-billion-token long-context phase that extends it to a 1-million-token context length
A 35-billion-token continued-pretraining stage extends Super to a 1-million-token context length.
Both models are trained for context lengths up to 1 million tokens.
Following distillation, both models undergo long-context training.
a 35B-token extension to 1M context
a 35B-token extension to 1M context