Open Models Almanac
Methodsmodel architecturepositional encoding

specific method · filed under model architecture

No Position Encoding

No Position Encoding (NoPE) omits any explicit positional encoding from a layer's queries and keys.1 It is used in hybrid designs where other layers supply position-sensitive sequence mixing, letting the NoPE layers handle unrestricted content interaction.2

Trade-off It avoids modifying positional-encoding parameters when extending the context length, such as retuning a RoPE frequency base.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. applies No Position Encoding (NoPE) to all MLA layers. Consequently, no explicit positional encoding is applied to their queries or keys — k3 tech report
  2. The intervening KDA layers provide position-sensitive and recency-aware sequence mixing, while the MLA layers provide unrestricted global content interaction — k3 tech report
  3. avoids modifying positional-encoding parameters when extending the context length, such as retuning a RoPE frequency base — k3 tech report

Also called No Position Encoding on MLA layers, No Position Encoding (NoPE), No positional embedding (NoPE), NoPE (no explicit positional embedding), No Positional Encoding (NoPE), do not use RoPE in attention layers.

sources
3
models
3
labs adopt it
3
strongest
core

In practice

Checked as “Layers without positional encoding (NoPE)” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
Mistral-Large-4in its codenot stated
no_rope_layers=[1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0…

no_rope_layers marks every 4th softmax-attention layer (value 0) to skip RoPE entirely while other layers apply rotary embeddings, all within the same indexed_attention layer type.

text_config.no_rope_layers = [1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1…
self.use_rope = config.no_rope_layers[layer_idx]
modeling_mistral_large4.py · L209
if self.use_rope:
            cos, sin = position_embeddings
            # For layers with cache sharing, only rotate the queries: the keys were already rotated by the source layer
            if self.is_kv_shared_layer:
                q_rot, _ = apply_rotary_pos_emb_interleave(q_rot, None, cos, sin)
modeling_mistral_large4.py · L425–429
Kolibri-1in its codeadopted
layer_types=["sliding_attention", "sliding_atte…

Full-attention layers skip rotary position embedding entirely (NoPE) while sliding-window layers apply RoPE, per the is_sliding check.

layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte…
        # We use global NoPE for hybrid attention model
        if self.sliding_window is None or self.is_sliding:
            query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
modeling_kolibri1.py · L181–183
LFM3-1.7Bnot in its codenot stated
rope_parameters={"full_attention": {"partial_rotary…

All attention layer types (full_attention, sliding_attention) have an entry in rope_parameters and receive rotary embeddings; only non-attention conv layers lack RoPE, which does not count.

rope_parameters = {"full_attention": {"partial_rotary_factor": 0.5, "rope_theta": 1000000.0, "r…
self.layer_types = [layer_type for layer_type in config._rope_type_labels if layer_type in config.layer_types]
modeling_lfm3.py · L72
MiniMax-M3.1-Flashnot in its codenot stated
verdict and lines

layer_types only toggles between full_attention and the sparse-indexer attention type; RoPE position_embeddings are passed to and applied in every decoder layer uniformly, with no layer lacking positional encoding.

cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
modeling_minimax_m3_vl.py · L435–436
DeepSeek-V4.1-Flash-1017code unclearnot stated
verdict and lines

No explicit no_rope_layers or nope_layer_interval config key is present, and the given attention implementation files do not show a per-layer switch disabling RoPE for softmax attention layers, so this cannot be confirmed from the provided code/config.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

No no_rope_layers or nope_layer_interval keys appear in the config, and the attention implementation deciding RoPE application per layer is not provided.

Step-5-Previewcode unclearnot stated
verdict and lines

Code supports a use_rope_layers list that can disable RoPE per layer, but this config does not set use_rope_layers, so whether any softmax-attention layer runs without RoPE cannot be determined.

self.use_rope = bool(
            _per_layer_value(
                use_rope_layers,
                self.layer_idx,
                name="use_rope_layers",
                default=True,
            )
        )
model.py · L1153–1160
MiMo-V2.6-Flashnot in its codenot stated
verdict and lines

Every attention layer (full or sliding-window) applies rotary embeddings via its own rotary embedding module; there is no layer that skips RoPE entirely for softmax attention.

MiMo-V2.6-Pronot in its codenot stated
verdict and lines

All attention layers (full and SWA) apply the same partial RoPE scheme uniformly; there is no layer-specific toggle disabling RoPE entirely for some softmax attention layers.

DeepSeek-V4.1-Flashcode unclearnot stated
verdict and lines

No explicit no_rope_layers/nope_layer_interval config key or code path is visible in the given files to determine whether some softmax attention layers skip positional encoding entirely.

Hy4-previewnot in its codenot stated
verdict and lines

Every attention layer applies rotary embeddings to its rope slice uniformly (via the shared HYV4RotaryEmbedding and apply_rotary_pos_emb call in every HYV4Attention forward); there is no config or code path that skips RoPE entirely for a subset of softmax attention layers.

GLM-5.3-Flashnot in its codenot stated
mla_use_nope=true · qk_rope_head_dim=0

All softmax-attention (indexed/DSA) layers uniformly use no positional encoding (qk_rope_head_dim=0, mla_use_nope=true, position_embeddings=None for every layer) — this is a uniform NoPE setup rather than some layers having RoPE and others not.

text_config.mla_use_nope = truetext_config.qk_rope_head_dim = 0
                # Key change using NoPE
                position_embeddings=None,
modeling_glm5_next.py · L1502–1503
Qwen3.8-Flash-Nextnot in its codenot stated
verdict and lines

All softmax-attention (indexed_attention/full_attention) layers apply the same partial RoPE via apply_rotary_pos_emb; there is no per-layer toggle disabling RoPE entirely on some attention layers (linear_attention layers are not softmax attention and don't count).

DeepSeek-V4-Flash-Vision-Expnot in its codenot stated
verdict and lines

Every attention layer applies rotary embedding (either main or compress rope type) on its trailing rope slice; there is no layer that skips positional encoding entirely, only partial RoPE per head.

self.rope_layer_type = "main" if self.layer_type == "sliding_attention" else "compress"
modeling_deepseek_v4.py · L788
GLM-5.3not in its codenot stated
verdict and lines

Every attention layer is the same GlmMoeDsaAttention applying rotary embeddings to the rope slice; there is no per-layer toggle disabling RoPE entirely for some softmax attention layers.

DeepSeek-V4-Pro-0813not in its codenot stated
verdict and lines

Every attention layer applies rotary position embedding (either main theta=10000 or compress theta=160000 with yarn) to its rope slice; there is no layer type that skips positional encoding entirely

self.rope_layer_type = "main" if self.layer_type == "sliding_attention" else "compress"
modeling_deepseek_v4.py · L788
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bcode unclearadopted
verdict and lines

The attention module's forward does not show rotary embedding application at all (no cos/sin passed into NemotronHAttention.forward), so it cannot be determined whether some full-attention layers skip positional encoding while others apply it.

def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: torch.Tensor | None = None,
        past_key_values: Cache | None = None,
        **kwargs: Unpack[TransformersKwargs],
    ) -> tuple[torch.Tensor, torch.Tensor | None]:
modeling_nemotron_h.py · L886–892
DeepSeek-V4-Flash-0731not in its codenot stated
verdict and lines

All attention layers apply partial RoPE to the trailing rope slice of every head; none of the softmax attention layers skip positional encoding entirely, so there is no NoPE/RoPE layer split.

Xiaomi-Robotics-1not in its codenot stated
verdict and lines

Every Qwen3VLTextDecoderLayer uses the same attention class with RoPE applied uniformly; there is no per-layer switch omitting positional encoding.

Laguna-S-2.1not in its codenot stated
rope_type=default

Every layer type (full_attention and sliding_attention) has a rope_parameters entry with a defined rope_type, so no softmax attention layer skips positional encoding entirely.

rope_parameters.sliding_attention.rope_type = "default"
for layer_type in self.layer_types:
            rope_params = self.config.rope_parameters[layer_type]
            if rope_params is None:
                continue
modeling_laguna.py · L74–77
Kimi K3not in its codeadopted
mla_use_nope=true

All full-attention (MLA) layers use mla_use_nope=True uniformly, meaning nope is applied to every softmax-attention layer's designated portion rather than some layers having positional encoding and others not; there is no layer-varying NoPE pattern.

text_config.mla_use_nope = true
self.use_nope = config.mla_use_nope
            self.scaling = self.q_head_dim ** (-0.5)
modeling_kimi_linear.py · L358–359
Inklingcode unclearnot stated
verdict and lines

The model has no RoPE at all (it uses a learned relative-position bias instead), so there is no distinction between layers with and without positional encoding visible in this code; this check does not clearly apply and the files don't show any per-layer toggle for positional encoding.

Hy3not in its codenot stated
verdict and lines

RoPE (via position_embeddings) is applied uniformly to every attention layer with no per-layer no-rope switch.

query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
modeling_hy_v3.py · L245
GLM-5.2not in its codenot stated
verdict and lines

Every attention layer applies the same decoupled RoPE scheme (rotary applied to the rope slice); there is no subset of softmax attention layers lacking positional encoding entirely.

NVIDIA-Nemotron-3-Ultra-550B-A55Bcode unclearnot stated
verdict and lines

NemotronHAttention never calls apply_rotary_pos_emb or references rotary embeddings in its forward, and no rotary embedding module is instantiated in the given files, so whether full_attention layers receive any positional encoding cannot be determined from this code.

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L896–898
MiniMax-M3not in its codenot stated
verdict and lines

All attention layers apply the same rotary embedding via apply_rotary_pos_emb; there is no per-layer toggle to skip positional encoding entirely.

Step-3.7-Flashnot in its codenot stated
use_rope_layers=[]

Every layer_type (full_attention and sliding_attention) gets an entry in rope_parameters with rope_type 'default' plus rope_theta, so all attention layers receive positional encoding; there is no no_rope_layers mechanism in the code.

text_config.use_rope_layers = []
params = {"rope_type": "default", "rope_theta": rope_theta[i]}
                if partial_rotary_factors:
                    params["partial_rotary_factor"] = partial_rotary_factors[i]
                self.rope_parameters[layer_type] = params
configuration_step3p7.py · L305–308
MolmoAct 2not in its codenot stated
rope_scaling_layers=null

Every decoder layer calls apply_rotary_pos_emb on position_embeddings unconditionally; rope_scaling_layers is null so there's no per-layer selection skipping RoPE on some attention layers.

text_config.rope_scaling_layers = null
cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(
            query_states, key_states, cos, sin
        )
modeling_molmoact2.py · L2285–2288
DeepSeek-V4-Flashnot in its codenot stated
qk_rope_head_dim=64

All attention layers apply partial RoPE to the same trailing slice of each head (the leading slice is always unrotated, not a whole layer being without any positional encoding), so there is no layer-level NoPE/RoPE split across whole softmax-attention layers.

qk_rope_head_dim = 64
DeepSeek-V4-Pronot in its codenot stated
verdict and lines

All attention layers apply partial RoPE (either 'main' or 'compress' rope type) to their rope slice; there is no attention layer that skips positional encoding entirely.

self.rope_layer_type = "main" if self.layer_type == "sliding_attention" else "compress"
modeling_deepseek_v4.py · L788
MiMo-V2.5not in its codenot stated
verdict and lines

All softmax attention layers apply RoPE to the rope_dim portion of every head (partial RoPE uniformly applied); there is no per-layer toggle disabling RoPE entirely on some attention layers.

MiMo-V2.5-Pronot in its codenot stated
verdict and lines

All softmax-attention layers apply the same partial_rotary_factor RoPE scheme uniformly (full-attention and sliding-window layers both get rotary embeddings via rotary_emb/swa_rotary_emb); there is no layer subset entirely without positional encoding.

position_embeddings = self.rotary_emb(hidden_states, position_ids)
        swa_position_embeddings = self.swa_rotary_emb(hidden_states, position_ids)
modeling_mimo_v2.py · L589–590
Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

All full_attention layers apply rotary embeddings uniformly via apply_rotary_pos_emb; there is no per-layer flag disabling RoPE for some softmax attention layers (linear-attention layers lacking RoPE don't count).

Gemma 4 31Bnot in its codenot stated
verdict and lines

Every unique layer_type (sliding_attention, full_attention) has an associated rope_parameters entry and the rotary embedding is applied unconditionally per layer type, so no attention layer lacks positional encoding.

for layer_type in self.layer_types:
            rope_params = self.config.rope_parameters[layer_type]
            if rope_params is None:
                continue
modeling_gemma4.py · L1089–1092
Qwen3.5-397B-A17Bnot in its codenot stated
verdict and lines

All full_attention layers apply rotary embeddings via position_embeddings/apply_rotary_pos_emb; there is no no_rope_layers config or per-layer skip of RoPE among attention layers.

DeepSeek-V3.2not in its codenot stated
verdict and lines

layer_types is uniformly 'indexed_attention' for all layers and every layer applies the same partial RoPE pattern via apply_rotary_pos_emb_interleave; no layer fully skips positional encoding.

if self.layer_types is None:
            self.layer_types = ["indexed_attention"] * self.num_hidden_layers
configuration_deepseek_v32.py · L138–139
X-VLAnot in its codenot stated
verdict and lines

All attention layers in the text model use the same learned absolute positional embedding scheme uniformly; there is no mixture of layers with and without positional encoding.

gpt-oss-120bnot in its codenot stated
verdict and lines

All attention layers receive the same rotary position embeddings computed once and passed to every layer; there is no layer-conditional skip of RoPE

cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
modeling_gpt_oss.py · L302–303
OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No no_rope_layers, nope_layer_interval, or layer_types config exists, and no code path skips positional encoding for some attention layers.

Documented in

Evidence

4 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 3

Kolibri-1

Aleph Alphaadopted1 source

the full-attention blocks use No Positional Encoding (NoPE)

coremodel architecturein KolibriAleph Alpha

Kimi K3

Moonshot AIadopted1 source

Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA.

coreunclearin Kimi K3Moonshot AI
1 more from this model

Kimi K3 follows the hybrid design of Kimi Linear [63] and applies No Position Encoding (NoPE) to all MLA layers.

usedmodel architecturein Kimi K3Moonshot AI
↑