specific method · filed under model architecture
No Position Encoding
A design that omits explicit positional embeddings, with positional information instead described as implicit in recurrent gating and decay.
Also called No Position Encoding on MLA layers, No Position Encoding (NoPE), No positional embedding (NoPE), NoPE (no explicit positional embedding).
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1core 1
Documented in
In the code
Checked as “Layers without positional encoding (NoPE)” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | in its code | not stated | All softmax-attention (MLA/deepseek_sparse_attention) layers apply no positional encoding at all (qk_rope_head_dim=0, mla_use_nope=true, and the model forward passes position_embeddings=None for every layer), which is a fixed NoPE choice rather than a mix of RoPE/no-RoPE layers, but it does mean the attention layers apply no rotary; since it's uniform across all attention layers (not some layers with and some without), this is effectively a fully NoPE model rather than the interleaved NoPE pattern described. modeling_glm5_next.py#L1502 |
| DeepSeek-V4.1-Flash | not in its code | not stated | There is no per-layer flag disabling RoPE entirely for some softmax attention layers; all layers apply RoPE to the rotary portion of Q/K (partial RoPE), and no no_rope_layers/nope_layer_interval mechanism is present. |
| Hy4-preview | not in its code | not stated | layer_types shows every layer is the same deepseek_sparse_attention type and all apply RoPE to their rope slice; there is no subset of full-attention layers that skip positional encoding entirely. |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | All attention layers apply partial RoPE to the trailing rope slice of every head (either 'main' or 'compress' rope type); no layer type is fully without positional encoding. modeling_deepseek_v4.py#L817 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | code unclear | not stated | The attention layer's forward pass does not show any RoPE application (no apply_rotary_pos_emb call visible in NemotronHAttention.forward), so it is unclear from the given code whether some full-attention layers omit position encoding while others include it. modeling_nemotron_h.py#L866 |
| MiMo-V2.6-Flash | not in its code | not stated | Every attention layer (full or sliding-window) applies rotary embeddings via its own rotary embedding module; there is no layer that skips RoPE entirely for softmax attention. |
| DeepSeek-V4-Flash | not in its code | not stated | Every attention layer applies RoPE to its rope slice (partial RoPE on the trailing qk_rope_head_dim channels) regardless of layer_type; there is no layer that skips positional encoding entirely. modeling_deepseek_v4.py#L817 |
| MiMo-V2.5 | not in its code | not stated | All softmax attention layers apply RoPE to the rope_dim portion of every head (partial RoPE uniformly applied); there is no per-layer toggle disabling RoPE entirely on some attention layers. |
| GLM-5.3 | not in its code | not stated | Every attention layer uses the same partial-RoPE MLA formulation (qk_rope_head_dim rotated, qk_nope_head_dim not); there is no config or code path that fully skips RoPE on some softmax-attention layers while applying it on others. |
| Hy3 | not in its code | not stated | Rotary position embeddings are applied uniformly to every attention layer via the shared rotary_emb module with no per-layer skip logic modeling_hy_v3.py#L256 |
| GLM-5.2 | not in its code | not stated | Every attention layer uses the same qk_nope/qk_rope split within MLA (partial RoPE per head), rather than some full-attention layers entirely lacking positional encoding while others have it; there is no no_rope_layers/nope_layer_interval config or logic. |
| MiniMax-M3 | not in its code | not stated | Every attention layer (whether full_attention or minimax_m3_sparse) receives the same rotary position_embeddings applied via apply_rotary_pos_emb; there is no per-layer toggle to skip RoPE. modeling_minimax_m3_vl.py#L452 |
| DeepSeek-V3.2 | not in its code | not stated | All attention layers apply the same decoupled RoPE scheme uniformly; there is no per-layer switch disabling RoPE entirely for some softmax attention layers. |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | All attention layers apply RoPE (either 'main' or 'compress' rope type) to their rope slice; there is no layer type that skips positional encoding entirely, only partial RoPE within a head. modeling_deepseek_v4.py#L780 |
| DeepSeek-V4-Pro | not in its code | not stated | Every attention layer applies partial RoPE to its rope slice (sliding layers use 'main' rope, CSA/HCA layers use 'compress' rope); no layer type skips positional encoding entirely for its softmax attention modeling_deepseek_v4.py#L780 |
| DeepSeek-V4-Pro-0813 | not in its code | not stated | All attention layers apply RoPE (partial, on the trailing rope slice) via either the main or compress rope type; no attention layer entirely skips positional encoding. |
| Gemma 4 31B | not in its code | not stated | Every layer type (full_attention and sliding_attention) has an entry in rope_parameters and receives rotary embeddings; no attention layer type is configured to skip RoPE entirely. modeling_gemma4.py#L1092 |
| Inkling | code unclear | not stated | The model has no RoPE at all (uses relative bias instead), so the NoPE-layers concept (some attention layers with RoPE, others without) does not clearly apply and cannot be resolved from the given code. |
| Kimi K3 | not in its code | core | All full-attention (MLA) layers use mla_use_nope=True uniformly, meaning nope is applied to every softmax-attention layer's designated portion rather than some layers having positional encoding and others not; there is no layer-varying NoPE pattern. modeling_kimi_linear.py#L358 |
| Laguna-S-2.1 | not in its code | not stated | Every layer type (full_attention and sliding_attention) has an entry in rope_parameters with a rope_type, so all softmax-attention layers receive positional encoding. modeling_laguna.py#L388 |
| MiMo-V2.5-Pro | not in its code | not stated | All softmax-attention layers apply the same partial_rotary_factor RoPE scheme uniformly (full-attention and sliding-window layers both get rotary embeddings via rotary_emb/swa_rotary_emb); there is no layer subset entirely without positional encoding. modeling_mimo_v2.py#L589 |
| MiMo-V2.6-Pro | not in its code | not stated | All attention layers (full and SWA) apply the same partial RoPE scheme uniformly; there is no layer-specific toggle disabling RoPE entirely for some softmax attention layers. |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | code unclear | not stated | No rotary embedding application (cos/sin) is visible inside NemotronHAttention.forward in this file, so it's unclear whether full_attention layers receive positional encoding uniformly or whether some lack it; the rotary embedding module itself is not included here. |
| Qwen3.5-397B-A17B | not in its code | not stated | All full_attention layers apply the same RoPE via position_embeddings; there is no no_rope_layers/nope_layer_interval config or per-layer skip of RoPE for softmax attention layers. |
| Qwen3.6-35B-A3B | not in its code | not stated | There is no no_rope_layers/nope_layer_interval mechanism; RoPE (partial) is applied uniformly to every full_attention layer via the shared rotary embedding. modeling_qwen3_5_moe.py#L800 |
| Qwen3.8-Flash-Next | not in its code | not stated | There is no per-layer no_rope_layers config or code path that skips RoPE entirely for some softmax attention layers while applying it to others; all full/indexed attention layers use the same partial RoPE. |
| Step-3.7-Flash | not in its code | not stated | Every layer_type (full_attention and sliding_attention) has a rope_parameters entry built and applied via Step3p7RotaryEmbedding; there is no layer type that skips RoPE entirely. modeling_step3p7.py#L906 |
| gpt-oss-120b | not in its code | not stated | Rotary position embeddings are applied uniformly to every attention layer's queries and keys; there is no layer-conditional skip of RoPE. modeling_gpt_oss.py#L310 |
Evidence
2 spans quoted from the sources, strongest treatment first.
Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA.
coreunclearin Kimi K3Moonshot AI
Kimi K3 follows the hybrid design of Kimi Linear [63] and applies No Position Encoding (NoPE) to all MLA layers.
usedmodel architecturein Kimi K3Moonshot AI
Filed alongside
Other methods under model architecture :: positional encoding.
YaRNRotary Position Embedding2D rotary position embeddingRoPE scalingLearned input-dependent relative position biasProportional Rotary Position EmbeddingRelative attention2D coordinate-based positional embeddingsGated attention with partial RoPEOmitting RoPE in attention layersPartial RoPEPer-layer-type rotary position scalesTemporal-Modality Rotary Position Embedding