specific method · filed under model architecture
Progressive context extension
A training approach that progressively increases the context window across stages, rather than using a fixed context length throughout.
Also called context window extension, progressive context extension curriculum.
- sources
- 2
- models
- 3
- labs adopt it
- 2
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
We achieve this through extending the context window progressively as training proceeds, following a four-stage curriculum. The window grows from 8K to 64K tokens during pre-training, and from 256K to 1M tokens during the cooldown phase.
during which the context window is progressively extended from 32K to 256K to 1M
Filed alongside
Other methods under model architecture :: context capacity.