general family · filed under model architecture
RoPE scaling
A general family of techniques for scaling rotary position embeddings to handle longer contexts; the evidence does not specify one particular scaling algorithm.
Also called RoPE scaling for context extension, RoPE scaling techniques, RoPE scaling for context extension (YaRN).
- sources
- 3
- models
- 2
- lab adopt it
- 1
- strongest
- optional
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Extending Context Window of Large Language Models via Positional Interpolation (Chen et al., 2023) paper arxiv.orgoriginal RoPE-scaling method for context extension
- YaRN: Efficient Context Window Extension of Large Language Models (Peng et al., 2023) paper arxiv.orgwidely-used improved RoPE scaling method
Evidence
3 spans quoted from the sources, strongest treatment first.
we recommend using RoPE scaling techniques to handle long texts effectively, e.g., YaRN.
we recommend using RoPE scaling techniques to handle long texts effectively
we recommend using RoPE scaling techniques to handle long texts effectively.
Filed alongside
Other methods under model architecture :: positional encoding.