Open Models Almanac
Methodsmodel architecturepositional encoding

specific method · filed under model architecture

Partial rotary position embeddings

RoPE is applied to only part of the relevant dimensions or head, leaving the remainder unrotated; the evidence gives different application contexts and proportions.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See positional encoding for the methods filed beside it.

Also called partial RoPE, Partial rotary positional embeddings.

sources
3
models
2
labs adopt it
2
strongest
core

In practice

Checked as “Partial RoPE” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
Mistral-Large-4in its codenot stated
qk_rope_head_dim=64 · qk_nope_head_dim=64

RoPE is applied only to the qk_rope_head_dim slice (64) of each 128-dim qk head while qk_nope_head_dim (64) carries no rotary signal, DeepSeek-style decoupled RoPE.

text_config.qk_rope_head_dim = 64text_config.qk_nope_head_dim = 64
q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_mistral_large4.py · L412
LFM3-1.7Bin its codeadopted
partial_rotary_factor=0.5

full_attention layers use partial_rotary_factor 0.5, applying RoPE to only half of the head dimension.

rope_parameters.full_attention.partial_rotary_factor = 0.5
MiniMax-M3.1-Flashin its codenot stated
partial_rotary_factor=0.5 · rotary_dim=64

RoPE is applied only to the first rotary_dim=64 of each 128-dim head (partial_rotary_factor 0.5), the rest passing through unrotated.

text_config.rotary_dim = 64text_config.partial_rotary_factor = 0.5
rotary_dim = cos.shape[-1]
    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
modeling_minimax_m3_vl.py · L375–377
DeepSeek-V4.1-Flash-1017in its codenot stated
qk_rope_head_dim=64 · head_dim=512

qk_rope_head_dim (64) is smaller than head_dim (512), indicating only part of each query/key head carries RoPE (DeepSeek-style decoupled RoPE key), consistent with the compress_rope_theta and rotary cos/sin cache usage seen in the code.

text_config.qk_rope_head_dim = 64text_config.head_dim = 512
attn.rotary_emb.cos_sin_cache,
        attn.swa_cache_layer.block_size,
dspark.py · L284–285
Step-5-Previewin its codenot stated
partial_rotary_factors=[1.0, 1.0, 1.0, 0.3333333333333333,…

partial_rotary_factors alternates between 1.0 and 0.333... per layer, so full_attention layers apply RoPE to only one third of the head dimension.

text_config.partial_rotary_factors = [1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1…
self.partial_rotary_factor = float(partial_rotary_factor)
        self.rotary_dim = int(self.head_dim * self.partial_rotary_factor)
model.py · L954–955
MiMo-V2.6-Flashin its codenot stated
partial_rotary_factor=0.334 · head_dim=192

Each head's dimension is split into a rotary part (rope_dim, sized by partial_rotary_factor 0.334) and a non-rotary part (head_dim - rope_dim) that carries no position signal.

partial_rotary_factor = 0.334head_dim = 192
self.rope_dim = int(self.head_dim * getattr(config, "partial_rotary_factor", 1.0))
modeling_mimo_v2.py · L253
query_rope, query_nope = query_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
        key_rope, key_nope = key_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
modeling_mimo_v2.py · L306–307
MiMo-V2.6-Proin its codenot stated
partial_rotary_factor=0.334 · rope_parameters.partial_rotary_factor=0.334

partial_rotary_factor of 0.334 makes rope_dim smaller than head_dim, so RoPE is applied only to a split portion of each query/key head with the rest (query_nope/key_nope) carrying no position signal.

partial_rotary_factor = 0.334rope_parameters.partial_rotary_factor = 0.334
query_rope, query_nope = query_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
        key_rope, key_nope = key_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
modeling_mimo_v2.py · L306–307
DeepSeek-V4.1-Flashin its codenot stated
qk_rope_head_dim=64 · head_dim=512

qk_rope_head_dim (64) is smaller than head_dim (512), indicating only part of each head carries RoPE (DeepSeek-style decoupled RoPE key), consistent with MLA's partial rotary key.

text_config.qk_rope_head_dim = 64text_config.head_dim = 512
Hy4-previewin its codenot stated
qk_rope_head_dim=64 · qk_nope_head_dim=192

RoPE is applied only to the qk_rope_head_dim slice of each query/key (64 of the 256 total qk_head_dim), the qk_nope_head_dim portion carries no rotary signal, matching MLA's decoupled RoPE key.

qk_rope_head_dim = 64qk_nope_head_dim = 192
q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_hy_v4.py · L412
q_rot, k_rot = apply_rotary_pos_emb(q_rot, k_rot, cos, sin)
modeling_hy_v4.py · L422
Qwen3.8-Flash-Nextin its codeadopted
text_config.partial_rotary_factor=0.25 · text_config.rope_parameters.partial_rotary_factor=0.25

partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest (q_nope/k_nope) concatenated back unrotated.

text_config.partial_rotary_factor = 0.25text_config.rope_parameters.partial_rotary_factor = 0.25
q_rope, q_nope = q[..., :rotary_dim], q[..., rotary_dim:]
    # Apply rotary embeddings on the first half or full tensor
    q_rope = (q_rope * cos) + (rotate_half(q_rope) * sin)
    # Concatenate back to full shape
    q_rotated = torch.cat([q_rope, q_nope], dim=-1)
modeling_qwen4_exp.py · L648–652
DeepSeek-V4-Flash-Vision-Expin its codenot stated
qk_rope_head_dim=64 · head_dim=512

Only the trailing qk_rope_head_dim=64 of the 512-dim head carries rotary position signal, computed via partial_rotary_factor=64/512, with the leading nope slice left untouched.

qk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L367–370
GLM-5.3in its codenot stated
qk_rope_head_dim=64 · qk_nope_head_dim=192 · qk_head_dim=256

Only qk_rope_head_dim=64 of the qk_head_dim=256 query/key dimension carries RoPE (q_rot/q_pass split), the decoupled MLA RoPE key.

qk_rope_head_dim = 64qk_nope_head_dim = 192qk_head_dim = 256
q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_glm_moe_dsa.py · L384
DeepSeek-V4-Pro-0813in its codenot stated
qk_rope_head_dim=64 · head_dim=512

RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each head_dim=512 dimension, leaving the leading nope channels untouched, matching MLA's decoupled rope key pattern

qk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L367–370
DeepSeek-V4-Flash-0731in its codenot stated
qk_rope_head_dim=64 · head_dim=512

RoPE is applied only to the trailing rope_head_dim slice (64 of 512) of each query/key head, with the leading nope channels left untouched.

qk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L367–370
Laguna-S-2.1in its codenot stated
rope_parameters.full_attention.partial_rotary_factor=0.5 · rope_parameters.sliding_attention.partial_rotary_factor=1.0

full_attention layers use partial_rotary_factor 0.5 (rotary applied to half of head_dim), while sliding_attention layers use 1.0 (full rotary).

rope_parameters.full_attention.partial_rotary_factor = 0.5rope_parameters.sliding_attention.partial_rotary_factor = 1.0
partial_rotary_factor = config.rope_parameters[layer_type].get("partial_rotary_factor", 1.0)
        head_dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
        dim = int(head_dim * partial_rotary_factor)
modeling_laguna.py · L106–108
Kimi K3in its codenot stated
qk_rope_head_dim=64 · qk_nope_head_dim=128 · mla_use_nope=true

MLA queries/keys are split into a nope portion (qk_nope_head_dim=128) and a rope portion (qk_rope_head_dim=64), with only the rope portion carrying positional info via the decoupled key, and mla_use_nope confirms this partial scheme is active.

text_config.qk_nope_head_dim = 128text_config.qk_rope_head_dim = 64text_config.mla_use_nope = true
q_pass, q_rot = torch.split(
            q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_kimi_linear.py · L423–424
k_pass = self.kv_b_proj(self.kv_a_layernorm(
            k_pass)).view(key_shape).transpose(1, 2)
        k_pass, value_states = torch.split(
            k_pass, [self.qk_nope_head_dim, self.v_head_dim], dim=-1)
modeling_kimi_linear.py · L430–433
GLM-5.2in its codenot stated
qk_rope_head_dim=64 · qk_nope_head_dim=192

Only the qk_rope_head_dim=64 slice of each 256-dim query/key head is rotated (q_rot/q_pass split), the remaining qk_nope_head_dim=192 carries no position signal, as in DeepSeek-style decoupled RoPE.

qk_rope_head_dim = 64qk_nope_head_dim = 192
q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_glm_moe_dsa.py · L384
MiniMax-M3in its codenot stated
partial_rotary_factor=0.5 · rotary_dim=64

partial_rotary_factor is 0.5 and apply_rotary_pos_emb only rotates the first rotary_dim channels, passing the rest through unrotated.

text_config.partial_rotary_factor = 0.5text_config.rotary_dim = 64
rotary_dim = cos.shape[-1]
    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
modeling_minimax_m3_vl.py · L375–377
Step-3.7-Flashin its codenot stated
partial_rotary_factors=[0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0,…

partial_rotary_factors alternates 0.5 for full_attention layers and 1.0 for sliding_attention layers, and the rope init function scales head_dim by this factor to compute the rotary dim.

text_config.partial_rotary_factors = [0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1…
partial_rotary_factor = config.rope_parameters[layer_type].get("partial_rotary_factor", 1.0)
        head_dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
        dim = int(head_dim * partial_rotary_factor)
modeling_step3p7.py · L476–478
DeepSeek-V4-Flashin its codenot stated
qk_rope_head_dim=64 · head_dim=512

RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each 512-dim head, leaving the rest ("nope") channels without position signal, controlled by partial_rotary_factor derived from qk_rope_head_dim/head_dim.

qk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L367–370
DeepSeek-V4-Proin its codenot stated
qk_rope_head_dim=64 · head_dim=512

RoPE is applied only to the trailing qk_rope_head_dim=64 of each 512-dim head (partial_rotary_factor=64/512), leaving the rest of the head dimension without position signal.

qk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L367–370
MiMo-V2.5in its codenot stated
partial_rotary_factor=0.334 · head_dim=192

partial_rotary_factor=0.334 makes rope_dim less than head_dim, and the attention forward splits query/key into rope and nope parts, applying RoPE only to the rope slice.

partial_rotary_factor = 0.334head_dim = 192
query_rope, query_nope = query_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
        key_rope, key_nope = key_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
modeling_mimo_v2.py · L306–307
MiMo-V2.5-Proin its codenot stated
partial_rotary_factor=0.334

Only rope_dim = head_dim * partial_rotary_factor (0.334) of each head is rotated, the remainder (query_nope/key_nope) carries no position signal.

partial_rotary_factor = 0.334
query_rope, query_nope = query_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
        key_rope, key_nope = key_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
modeling_mimo_v2.py · L301–302
Qwen3.6-35B-A3Bin its codenot stated
text_config.partial_rotary_factor=0.25 · text_config.rope_parameters.partial_rotary_factor=0.25

partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest passed through unrotated (q_pass/k_pass).

text_config.partial_rotary_factor = 0.25text_config.rope_parameters.partial_rotary_factor = 0.25
rotary_dim = cos.shape[-1]
    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
modeling_qwen3_5_moe.py · L680–682
Gemma 4 31Bin its codenot stated
partial_rotary_factor=0.25

Full-attention layers' rope_parameters set partial_rotary_factor 0.25, applying rotary embedding to only a quarter of the head dimension for that layer type.

text_config.rope_parameters.full_attention.partial_rotary_factor = 0.25
base = config.rope_parameters[layer_type]["rope_theta"]
        dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
modeling_gemma4.py · L1128–1129
Qwen3.5-397B-A17Bin its codenot stated
rope_parameters={"mrope_interleaved": true, "mrope_…

partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the remainder passed through unrotated in apply_rotary_pos_emb.

text_config.rope_parameters = {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "defa…
rotary_dim = cos.shape[-1]
    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
modeling_qwen3_5_moe.py · L680–682
DeepSeek-V3.2in its codenot stated
qk_rope_head_dim=64 · qk_nope_head_dim=128

Only qk_rope_head_dim=64 of each head's dimension carries RoPE while qk_nope_head_dim=128 carries no position signal, both in the main attention and the decoupled MLA key.

qk_rope_head_dim = 64qk_nope_head_dim = 128
q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_deepseek_v32.py · L416
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

No partial_rotary_factor, qk_rope_head_dim, or rotary_dim keys appear in the text_config, and the attention/rope code is not included, so partial RoPE cannot be determined.

Kolibri-1not in its codenot stated
verdict and lines

RoPE is computed over the full head_dim (rotate_half splits the whole dimension in half for rotation, not a partial subset), with no partial_rotary_factor or rope_dim config present.

        base = config.rope_parameters["rope_theta"]
        dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
modeling_kolibri1.py · L423–424
GLM-5.3-Flashnot in its codenot stated
qk_rope_head_dim=0 · mla_use_nope=true

qk_rope_head_dim is 0 and mla_use_nope is true, so no rotary position encoding is applied to any part of the query/key heads in this config (position_embeddings is explicitly passed as None).

text_config.qk_rope_head_dim = 0text_config.mla_use_nope = true
                # Key change using NoPE
                position_embeddings=None,
modeling_glm5_next.py · L1502–1503
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bcode unclearnot stated
partial_rotary_factor=1.0

partial_rotary_factor is 1.0 (full rotary) but the attention forward code does not actually call apply_rotary_pos_emb or build rotary embeddings in the given excerpt, so it's unclear whether/how rotary is applied at all for this model's attention layers.

partial_rotary_factor = 1.0
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

RoPE is applied to the full head dimension via rotate_half over the whole head_dim with no partial_rotary_factor or split dimension; no mechanism reduces the rotated portion.

def rotate_half(x):
    """Rotates half the hidden dims of the input."""
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]
    return torch.cat((-x2, x1), dim=-1)
modeling_mibot.py · L122–126
Inklingnot in its codenot stated
verdict and lines

The model uses a learned relative-position bias (d_rel/rel_extent projected logits added to attention scores) rather than rotary position embeddings at all, so there is no partial-RoPE mechanism; no rotary application appears anywhere in the attention code.

Hy3not in its codenot stated
verdict and lines

RoPE is computed over the full head_dim (dim = head_dim) and applied to the whole query/key vectors, with no partial_rotary_factor or rope_dim splitting.

dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
modeling_hy_v3.py · L94
NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
partial_rotary_factor=1.0

partial_rotary_factor is 1.0, meaning RoPE is applied to the full head dimension, and no decoupled MLA-style rope key exists.

partial_rotary_factor = 1.0
MolmoAct 2not in its codenot stated
verdict and lines

RoPE is applied to the full head_dim (rotate_half splits the whole dimension in half for rotation, not a fractional sub-range), and there is no partial_rotary_factor or qk_rope_head_dim config key.

def rotate_half(x):
    """Rotates half the hidden dims of the input."""
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]
    return torch.cat((-x2, x1), dim=-1)
modeling_molmoact2.py · L1934–1938
X-VLAnot in its codenot stated
verdict and lines

No rotary position embedding is applied at all (text model uses learned absolute positional embeddings), so there is no partial RoPE mechanism.

self.embed_positions = Florence2LearnedPositionalEmbedding(
            config.max_position_embeddings,
            embed_dim,
        )
modeling_florence2.py · L1486–1489
gpt-oss-120bnot in its codenot stated
verdict and lines

Rotary embedding is applied via chunking the full head dimension in half (first_half/second_half), covering the entire head_dim rather than a partial slice

first_half, second_half = torch.chunk(x, 2, dim=-1)
    first_ = first_half * cos - second_half * sin
    second_ = second_half * cos + first_half * sin
    return torch.cat((first_, second_), dim=-1)
modeling_gpt_oss.py · L212–215
OpenVLA / OpenVLA-OFTcode unclearnot stated
model_type=llama

The language model backbone is standard Llama-2, whose RoPE implementation is not included in these files, and no partial_rotary_factor or qk_rope_head_dim key appears in text_config to confirm or deny partial RoPE.

text_config.model_type = "llama"

Documented in

Liquid AI
Alibaba (Qwen)

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 2
↑