Open Models Almanac
Methodsmodel architecturecontext capacity

specific method · filed under model architecture

Progressive context extension

A training approach that increases the context window across stages as training proceeds.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See context capacity for the methods filed beside it.

Also called context window extension, progressive context extension curriculum.

sources
2
models
3
labs adopt it
2
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2

Kimi K3

Moonshot AIadopted1 source

We achieve this through extending the context window progressively as training proceeds, following a four-stage curriculum. The window grows from 8K to 64K tokens during pre-training, and from 256K to 1M tokens during the cooldown phase.

usedunclearin Kimi K3Moonshot AI
↑