implementation detail · filed under model architecture
Partial RoPE
A configuration that applies rotary position encoding to only part of the represented dimensions; the cited indexer applies it to 64 of 128 dimensions.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1
Documented in
In the code
Checked as “Partial RoPE” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | not in its code | not stated | qk_rope_head_dim is 0 and mla_use_nope is true, and the config's validate_architecture raises if qk_rope_head_dim > 0, so the decoupled RoPE key is disabled; text decoder layers pass position_embeddings=None (NoPE) rather than partial RoPE. configuration_glm5_next.py#L224 |
| DeepSeek-V4.1-Flash | in its code | not stated | Only qk_rope_head_dim (64) of the 512-dim head carries RoPE (nope_head_dim = head_dim - rope_head_dim), a decoupled/partial RoPE key as in MLA. attention.py#L288 |
| Hy4-preview | in its code | not stated | RoPE is applied only to the qk_rope_head_dim slice (64 of 256) of each query/key head, with the remaining qk_nope_head_dim portion carrying no position signal, MLA decoupled-RoPE style. modeling_hy_v4.py#L424 |
| DeepSeek-V4-Flash-0731 | in its code | not stated | Only a trailing rope_head_dim slice (qk_rope_head_dim=64 out of head_dim=512) of each head receives rotary embeddings, the rest (nope) is left untouched. modeling_deepseek_v4.py#L359 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | code unclear | not stated | partial_rotary_factor is set to 1.0 (full rotary) but the attention forward code shown does not actually call apply_rotary_pos_emb or reference rotary embeddings at all, making the RoPE application path itself unclear from the given files. |
| MiMo-V2.6-Flash | in its code | not stated | Each head's dimension is split into a rotary part (rope_dim, sized by partial_rotary_factor 0.334) and a non-rotary part (head_dim - rope_dim) that carries no position signal. modeling_mimo_v2.py#L253 |
| DeepSeek-V4-Flash | in its code | not stated | RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each 512-dim head, with the leading nope portion left untouched, per partial_rotary_factor. modeling_deepseek_v4.py#L359 |
| MiMo-V2.5 | in its code | not stated | partial_rotary_factor=0.334 makes rope_dim less than head_dim, and the attention forward splits query/key into rope and nope parts, applying RoPE only to the rope slice. modeling_mimo_v2.py#L306 |
| GLM-5.3 | in its code | not stated | RoPE is applied only to the qk_rope_head_dim slice (64 of 256 total qk_head_dim) while qk_nope_head_dim carries no positional signal, the MLA decoupled-RoPE-key pattern. modeling_glm_moe_dsa.py#L396 |
| Hy3 | not in its code | not stated | RoPE is computed and applied over the full head dimension (dim = head_dim, cos/sin concatenated over full size) with no partial rotary factor or split modeling_hy_v3.py#L97 |
| GLM-5.2 | in its code | not stated | Query/key heads split into a nope part (qk_nope_head_dim=192) with no rotation and a rope part (qk_rope_head_dim=64) that gets RoPE, MLA's decoupled RoPE key style. modeling_glm_moe_dsa.py#L396 |
| MiniMax-M3 | in its code | not stated | partial_rotary_factor is 0.5 and apply_rotary_pos_emb only rotates the first rotary_dim channels of each head, leaving the remainder untouched. modeling_minimax_m3_vl.py#L392 |
| DeepSeek-V3.2 | in its code | not stated | RoPE is applied only to the qk_rope_head_dim slice (64) of each 192-dim query/key head, while qk_nope_head_dim (128) carries no positional signal — DeepSeek's decoupled RoPE key. modeling_deepseek_v32.py#L430 |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | RoPE is applied only to the trailing qk_rope_head_dim=64 of the 512-dim head, leaving the leading nope channels untouched. modeling_deepseek_v4.py#L360 |
| DeepSeek-V4-Pro | in its code | not stated | RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each 512-dim head, leaving the leading nope portion unrotated modeling_deepseek_v4.py#L359 |
| DeepSeek-V4-Pro-0813 | in its code | not stated | RoPE is applied only to the trailing rope_head_dim slice of each head (qk_rope_head_dim=64 out of head_dim=512), leaving the leading nope channels untouched. modeling_deepseek_v4.py#L359 |
| Gemma 4 31B | in its code | not stated | Full-attention layers use a partial_rotary_factor of 0.25 in rope_parameters, applying RoPE to only a quarter of the head dimension. configuration_gemma4.py#L205 |
| Inkling | not in its code | not stated | No RoPE is applied at all in this attention implementation (position signal comes from a relative-bias projection r_out/rel_logits, not rotary embeddings), so there is no partial-RoPE mechanism. attention.py#L133 |
| Kimi K3 | in its code | not stated | MLA queries/keys are split into a nope portion (qk_nope_head_dim=128) and a rope portion (qk_rope_head_dim=64), with only the rope portion carrying positional info via the decoupled key, and mla_use_nope confirms this partial scheme is active. modeling_kimi_linear.py#L423 |
| Laguna-S-2.1 | in its code | not stated | Full-attention layers use partial_rotary_factor 0.5 so RoPE is applied to only half of each head dimension. modeling_laguna.py#L109 |
| MiMo-V2.5-Pro | in its code | not stated | Only rope_dim = head_dim * partial_rotary_factor (0.334) of each head is rotated, the remainder (query_nope/key_nope) carries no position signal. modeling_mimo_v2.py#L301 |
| MiMo-V2.6-Pro | in its code | not stated | partial_rotary_factor of 0.334 makes rope_dim smaller than head_dim, so RoPE is applied only to a split portion of each query/key head with the rest (query_nope/key_nope) carrying no position signal. modeling_mimo_v2.py#L306 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | partial_rotary_factor is 1.0, and no rotary embedding application is even visible in NemotronHAttention.forward (no cos/sin/apply_rotary_pos_emb call), so full or no RoPE is applied uniformly with no partial split. |
| Qwen3.5-397B-A17B | in its code | not stated | partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest passed through unrotated in apply_rotary_pos_emb. modeling_qwen3_5_moe.py#L179 |
| Qwen3.6-35B-A3B | in its code | not stated | partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest passed through unrotated in apply_rotary_pos_emb. modeling_qwen3_5_moe.py#L700 |
| Qwen3.8-Flash-Next | in its code | used | partial_rotary_factor of 0.25 means RoPE is applied only to a quarter of the 256-dim head, the rest carrying no position signal via rotate/nope split. modeling_qwen4_exp.py#L658 |
| Step-3.7-Flash | in its code | not stated | partial_rotary_factors is 0.5 for full_attention layers, and compute_default_rope_parameters scales the rotary dim by partial_rotary_factor, applying rotation to only part of the head. modeling_step3p7.py#L478 |
| gpt-oss-120b | not in its code | not stated | Rotary embedding is applied to the full head dimension via chunking into two halves, not a partial slice. modeling_gpt_oss.py#L220 |
Evidence
1 span quoted from the sources, strongest treatment first.
partial RoPE is applied to 64 of the 128 dimensions in each indexer head
usedunclearin Qwen3.8-Flash-NextQwen
Filed alongside
Other methods under model architecture :: positional encoding.
YaRNRotary Position Embedding2D rotary position embeddingRoPE scalingLearned input-dependent relative position biasNo Position EncodingProportional Rotary Position EmbeddingRelative attention2D coordinate-based positional embeddingsGated attention with partial RoPEOmitting RoPE in attention layersPer-layer-type rotary position scalesTemporal-Modality Rotary Position Embedding