LFM3-1.7B
Liquid AIGlobal attention layers rotate only the highest-frequency half of each head and leave the low-frequency half unrotated
1 more from this model
global attention layers rotate only the highest-frequency half of each head
specific method · filed under model architecture
RoPE is applied to only part of the relevant dimensions or head, leaving the remainder unrotated; the evidence gives different application contexts and proportions.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See positional encoding for the methods filed beside it.
Also called partial RoPE, Partial rotary positional embeddings.
Checked as “Partial RoPE” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| Mistral-Large-4 | in its code | not stated | qk_rope_head_dim=64 · qk_nope_head_dim=64RoPE is applied only to the qk_rope_head_dim slice (64) of each 128-dim qk head while qk_nope_head_dim (64) carries no rotary signal, DeepSeek-style decoupled RoPE. text_config.qk_rope_head_dim = 64text_config.qk_nope_head_dim = 64 | code → |
| LFM3-1.7B | in its code | adopted | partial_rotary_factor=0.5full_attention layers use partial_rotary_factor 0.5, applying RoPE to only half of the head dimension. rope_parameters.full_attention.partial_rotary_factor = 0.5 | code → |
| MiniMax-M3.1-Flash | in its code | not stated | partial_rotary_factor=0.5 · rotary_dim=64RoPE is applied only to the first rotary_dim=64 of each 128-dim head (partial_rotary_factor 0.5), the rest passing through unrotated. text_config.rotary_dim = 64text_config.partial_rotary_factor = 0.5 | code → |
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | qk_rope_head_dim=64 · head_dim=512qk_rope_head_dim (64) is smaller than head_dim (512), indicating only part of each query/key head carries RoPE (DeepSeek-style decoupled RoPE key), consistent with the compress_rope_theta and rotary cos/sin cache usage seen in the code. text_config.qk_rope_head_dim = 64text_config.head_dim = 512 | code → |
| Step-5-Preview | in its code | not stated | partial_rotary_factors=[1.0, 1.0, 1.0, 0.3333333333333333,…partial_rotary_factors alternates between 1.0 and 0.333... per layer, so full_attention layers apply RoPE to only one third of the head dimension. text_config.partial_rotary_factors = [1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1… | code → |
| MiMo-V2.6-Flash | in its code | not stated | partial_rotary_factor=0.334 · head_dim=192Each head's dimension is split into a rotary part (rope_dim, sized by partial_rotary_factor 0.334) and a non-rotary part (head_dim - rope_dim) that carries no position signal. partial_rotary_factor = 0.334head_dim = 192 | code → |
| MiMo-V2.6-Pro | in its code | not stated | partial_rotary_factor=0.334 · rope_parameters.partial_rotary_factor=0.334partial_rotary_factor of 0.334 makes rope_dim smaller than head_dim, so RoPE is applied only to a split portion of each query/key head with the rest (query_nope/key_nope) carrying no position signal. partial_rotary_factor = 0.334rope_parameters.partial_rotary_factor = 0.334 | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | qk_rope_head_dim=64 · head_dim=512qk_rope_head_dim (64) is smaller than head_dim (512), indicating only part of each head carries RoPE (DeepSeek-style decoupled RoPE key), consistent with MLA's partial rotary key. text_config.qk_rope_head_dim = 64text_config.head_dim = 512 | code → |
| Hy4-preview | in its code | not stated | qk_rope_head_dim=64 · qk_nope_head_dim=192RoPE is applied only to the qk_rope_head_dim slice of each query/key (64 of the 256 total qk_head_dim), the qk_nope_head_dim portion carries no rotary signal, matching MLA's decoupled RoPE key. qk_rope_head_dim = 64qk_nope_head_dim = 192 | code → |
| Qwen3.8-Flash-Next | in its code | adopted | text_config.partial_rotary_factor=0.25 · text_config.rope_parameters.partial_rotary_factor=0.25partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest (q_nope/k_nope) concatenated back unrotated. text_config.partial_rotary_factor = 0.25text_config.rope_parameters.partial_rotary_factor = 0.25 | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | qk_rope_head_dim=64 · head_dim=512Only the trailing qk_rope_head_dim=64 of the 512-dim head carries rotary position signal, computed via partial_rotary_factor=64/512, with the leading nope slice left untouched. qk_rope_head_dim = 64head_dim = 512 | code → |
| GLM-5.3 | in its code | not stated | qk_rope_head_dim=64 · qk_nope_head_dim=192 · qk_head_dim=256Only qk_rope_head_dim=64 of the qk_head_dim=256 query/key dimension carries RoPE (q_rot/q_pass split), the decoupled MLA RoPE key. qk_rope_head_dim = 64qk_nope_head_dim = 192qk_head_dim = 256 | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | qk_rope_head_dim=64 · head_dim=512RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each head_dim=512 dimension, leaving the leading nope channels untouched, matching MLA's decoupled rope key pattern qk_rope_head_dim = 64head_dim = 512 | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | qk_rope_head_dim=64 · head_dim=512RoPE is applied only to the trailing rope_head_dim slice (64 of 512) of each query/key head, with the leading nope channels left untouched. qk_rope_head_dim = 64head_dim = 512 | code → |
| Laguna-S-2.1 | in its code | not stated | rope_parameters.full_attention.partial_rotary_factor=0.5 · rope_parameters.sliding_attention.partial_rotary_factor=1.0full_attention layers use partial_rotary_factor 0.5 (rotary applied to half of head_dim), while sliding_attention layers use 1.0 (full rotary). rope_parameters.full_attention.partial_rotary_factor = 0.5rope_parameters.sliding_attention.partial_rotary_factor = 1.0 | code → |
| Kimi K3 | in its code | not stated | qk_rope_head_dim=64 · qk_nope_head_dim=128 · mla_use_nope=trueMLA queries/keys are split into a nope portion (qk_nope_head_dim=128) and a rope portion (qk_rope_head_dim=64), with only the rope portion carrying positional info via the decoupled key, and mla_use_nope confirms this partial scheme is active. text_config.qk_nope_head_dim = 128text_config.qk_rope_head_dim = 64text_config.mla_use_nope = true | code → |
| GLM-5.2 | in its code | not stated | qk_rope_head_dim=64 · qk_nope_head_dim=192Only the qk_rope_head_dim=64 slice of each 256-dim query/key head is rotated (q_rot/q_pass split), the remaining qk_nope_head_dim=192 carries no position signal, as in DeepSeek-style decoupled RoPE. qk_rope_head_dim = 64qk_nope_head_dim = 192 | code → |
| MiniMax-M3 | in its code | not stated | partial_rotary_factor=0.5 · rotary_dim=64partial_rotary_factor is 0.5 and apply_rotary_pos_emb only rotates the first rotary_dim channels, passing the rest through unrotated. text_config.partial_rotary_factor = 0.5text_config.rotary_dim = 64 | code → |
| Step-3.7-Flash | in its code | not stated | partial_rotary_factors=[0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0,…partial_rotary_factors alternates 0.5 for full_attention layers and 1.0 for sliding_attention layers, and the rope init function scales head_dim by this factor to compute the rotary dim. text_config.partial_rotary_factors = [0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1… | code → |
| DeepSeek-V4-Flash | in its code | not stated | qk_rope_head_dim=64 · head_dim=512RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each 512-dim head, leaving the rest ("nope") channels without position signal, controlled by partial_rotary_factor derived from qk_rope_head_dim/head_dim. qk_rope_head_dim = 64head_dim = 512 | code → |
| DeepSeek-V4-Pro | in its code | not stated | qk_rope_head_dim=64 · head_dim=512RoPE is applied only to the trailing qk_rope_head_dim=64 of each 512-dim head (partial_rotary_factor=64/512), leaving the rest of the head dimension without position signal. qk_rope_head_dim = 64head_dim = 512 | code → |
| MiMo-V2.5 | in its code | not stated | partial_rotary_factor=0.334 · head_dim=192partial_rotary_factor=0.334 makes rope_dim less than head_dim, and the attention forward splits query/key into rope and nope parts, applying RoPE only to the rope slice. partial_rotary_factor = 0.334head_dim = 192 | code → |
| MiMo-V2.5-Pro | in its code | not stated | partial_rotary_factor=0.334Only rope_dim = head_dim * partial_rotary_factor (0.334) of each head is rotated, the remainder (query_nope/key_nope) carries no position signal. partial_rotary_factor = 0.334 | code → |
| Qwen3.6-35B-A3B | in its code | not stated | text_config.partial_rotary_factor=0.25 · text_config.rope_parameters.partial_rotary_factor=0.25partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest passed through unrotated (q_pass/k_pass). text_config.partial_rotary_factor = 0.25text_config.rope_parameters.partial_rotary_factor = 0.25 | code → |
| Gemma 4 31B | in its code | not stated | partial_rotary_factor=0.25Full-attention layers' rope_parameters set partial_rotary_factor 0.25, applying rotary embedding to only a quarter of the head dimension for that layer type. text_config.rope_parameters.full_attention.partial_rotary_factor = 0.25 | code → |
| Qwen3.5-397B-A17B | in its code | not stated | rope_parameters={"mrope_interleaved": true, "mrope_…partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the remainder passed through unrotated in apply_rotary_pos_emb. text_config.rope_parameters = {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "defa… | code → |
| DeepSeek-V3.2 | in its code | not stated | qk_rope_head_dim=64 · qk_nope_head_dim=128Only qk_rope_head_dim=64 of each head's dimension carries RoPE while qk_nope_head_dim=128 carries no position signal, both in the main attention and the decoupled MLA key. qk_rope_head_dim = 64qk_nope_head_dim = 128 | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and linesNo partial_rotary_factor, qk_rope_head_dim, or rotary_dim keys appear in the text_config, and the attention/rope code is not included, so partial RoPE cannot be determined. | code → |
| Kolibri-1 | not in its code | not stated | verdict and linesRoPE is computed over the full head_dim (rotate_half splits the whole dimension in half for rotation, not a partial subset), with no partial_rotary_factor or rope_dim config present. | code → |
| GLM-5.3-Flash | not in its code | not stated | qk_rope_head_dim=0 · mla_use_nope=trueqk_rope_head_dim is 0 and mla_use_nope is true, so no rotary position encoding is applied to any part of the query/key heads in this config (position_embeddings is explicitly passed as None). text_config.qk_rope_head_dim = 0text_config.mla_use_nope = true | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | code unclear | not stated | partial_rotary_factor=1.0partial_rotary_factor is 1.0 (full rotary) but the attention forward code does not actually call apply_rotary_pos_emb or build rotary embeddings in the given excerpt, so it's unclear whether/how rotary is applied at all for this model's attention layers. partial_rotary_factor = 1.0 | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesRoPE is applied to the full head dimension via rotate_half over the whole head_dim with no partial_rotary_factor or split dimension; no mechanism reduces the rotated portion. | code → |
| Inkling | not in its code | not stated | verdict and linesThe model uses a learned relative-position bias (d_rel/rel_extent projected logits added to attention scores) rather than rotary position embeddings at all, so there is no partial-RoPE mechanism; no rotary application appears anywhere in the attention code. | code → |
| Hy3 | not in its code | not stated | verdict and linesRoPE is computed over the full head_dim (dim = head_dim) and applied to the whole query/key vectors, with no partial_rotary_factor or rope_dim splitting. | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | partial_rotary_factor=1.0partial_rotary_factor is 1.0, meaning RoPE is applied to the full head dimension, and no decoupled MLA-style rope key exists. partial_rotary_factor = 1.0 | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesRoPE is applied to the full head_dim (rotate_half splits the whole dimension in half for rotation, not a fractional sub-range), and there is no partial_rotary_factor or qk_rope_head_dim config key. | code → |
| X-VLA | not in its code | not stated | verdict and linesNo rotary position embedding is applied at all (text model uses learned absolute positional embeddings), so there is no partial RoPE mechanism. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesRotary embedding is applied via chunking the full head dimension in half (first_half/second_half), covering the entire head_dim rather than a partial slice | code → |
| OpenVLA / OpenVLA-OFT | code unclear | not stated | model_type=llamaThe language model backbone is standard Llama-2, whose RoPE implementation is not included in these files, and no partial_rotary_factor or qk_rope_head_dim key appears in text_config to confirm or deny partial RoPE. text_config.model_type = "llama" | code → |
3 quotes from 3 sources, one card per model, each opening with its strongest.
Global attention layers rotate only the highest-frequency half of each head and leave the low-frequency half unrotated
global attention layers rotate only the highest-frequency half of each head
partial RoPE is applied to 64 of the 128 dimensions in each indexer head