Open Models Almanac
Methodsmodel architecturepositional encoding

specific method · filed under model architecture

Rotary Position Embedding

Rotary Position Embedding (RoPE) is a positional encoding that encodes positional information into attention queries and keys.1,2 Because it is applied to both queries and KV entries, attention outputs carry relative position embeddings, so each entry's contribution depends on its distance to the query.3

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. we apply RoPE to its last 64 dimensions — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. We employ Rotary Positional Encodings (RoPE) [81] to encode positional information — laguna m1 xs2 technical report
  3. the output of the core attention will also carry relative position embeddings — the contribution of each KV entry to the core attention outputs will also be related to the distance between the query and the KV entry — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Also called RoPE, Rotary Positional Embedding (RoPE), rotary position embeddings (RoPE), rotary position embeddings, Rotary Positional Embedding, Rotary positional embeddings.

sources
6
models
7
labs adopt it
6
strongest
core

Documented in

Allen Institute for AI
MiniMax
Poolside
Xiaomi

Evidence

6 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
used 5core 1

MolmoAct 2

Allen Institute for AIadopted1 source

The expert uses rotary position embeddings for the action sequence, which gives the denoising transformer an explicit ordering over the predicted trajectory steps.

coreunclearin MolmoAct2 action expertAllen Institute for Artificial Intelligence (AI2)
↑