specific method · filed under model architecture
Rotary Position Embedding
A positional encoding that applies rotary transformations to attention representations; the supplied mentions also describe partial application in particular configurations.
Also called RoPE, Rotary Positional Embedding (RoPE), rotary position embeddings (RoPE), rotary position embeddings, Rotary Positional Embedding, Rotary positional embeddings.
- sources
- 5
- models
- 6
- labs adopt it
- 5
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
5 spans quoted from the sources, strongest treatment first.
We apply rotary position embeddings [14]
RoPE dimension 64
we partially employ the Rotary Positional Embedding (RoPE) to the attention queries, KV entries, and the core attention outputs
We employ Rotary Positional Encodings (RoPE) to encode positional information.
Rotary Positional Embedding (RoPE) is partially applied to the first 64 dimensions query and key
Filed alongside
Other methods under model architecture :: positional encoding.