Kolibri-1
Aleph Alphathe full-attention blocks use No Positional Encoding (NoPE)
specific method · filed under model architecture
No Position Encoding (NoPE) omits any explicit positional encoding from a layer's queries and keys.1 It is used in hybrid designs where other layers supply position-sensitive sequence mixing, letting the NoPE layers handle unrestricted content interaction.2
Trade-off It avoids modifying positional-encoding parameters when extending the context length, such as retuning a RoPE frequency base.3
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
applies No Position Encoding (NoPE) to all MLA layers. Consequently, no explicit positional encoding is applied to their queries or keys— k3 tech report
The intervening KDA layers provide position-sensitive and recency-aware sequence mixing, while the MLA layers provide unrestricted global content interaction— k3 tech report
avoids modifying positional-encoding parameters when extending the context length, such as retuning a RoPE frequency base— k3 tech report
Also called No Position Encoding on MLA layers, No Position Encoding (NoPE), No positional embedding (NoPE), NoPE (no explicit positional embedding), No Positional Encoding (NoPE), do not use RoPE in attention layers.
Checked as “Layers without positional encoding (NoPE)” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| Mistral-Large-4 | in its code | not stated | no_rope_layers=[1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0…no_rope_layers marks every 4th softmax-attention layer (value 0) to skip RoPE entirely while other layers apply rotary embeddings, all within the same indexed_attention layer type. text_config.no_rope_layers = [1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1… | code → |
| Kolibri-1 | in its code | adopted | layer_types=["sliding_attention", "sliding_atte…Full-attention layers skip rotary position embedding entirely (NoPE) while sliding-window layers apply RoPE, per the is_sliding check. layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte… | code → |
| LFM3-1.7B | not in its code | not stated | rope_parameters={"full_attention": {"partial_rotary…All attention layer types (full_attention, sliding_attention) have an entry in rope_parameters and receive rotary embeddings; only non-attention conv layers lack RoPE, which does not count. rope_parameters = {"full_attention": {"partial_rotary_factor": 0.5, "rope_theta": 1000000.0, "r… | code → |
| MiniMax-M3.1-Flash | not in its code | not stated | verdict and lineslayer_types only toggles between full_attention and the sparse-indexer attention type; RoPE position_embeddings are passed to and applied in every decoder layer uniformly, with no layer lacking positional encoding. | code → |
| DeepSeek-V4.1-Flash-1017 | code unclear | not stated | verdict and linesNo explicit no_rope_layers or nope_layer_interval config key is present, and the given attention implementation files do not show a per-layer switch disabling RoPE for softmax attention layers, so this cannot be confirmed from the provided code/config. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and linesNo no_rope_layers or nope_layer_interval keys appear in the config, and the attention implementation deciding RoPE application per layer is not provided. | code → |
| Step-5-Preview | code unclear | not stated | verdict and linesCode supports a use_rope_layers list that can disable RoPE per layer, but this config does not set use_rope_layers, so whether any softmax-attention layer runs without RoPE cannot be determined. | code → |
| MiMo-V2.6-Flash | not in its code | not stated | verdict and linesEvery attention layer (full or sliding-window) applies rotary embeddings via its own rotary embedding module; there is no layer that skips RoPE entirely for softmax attention. | code → |
| MiMo-V2.6-Pro | not in its code | not stated | verdict and linesAll attention layers (full and SWA) apply the same partial RoPE scheme uniformly; there is no layer-specific toggle disabling RoPE entirely for some softmax attention layers. | code → |
| DeepSeek-V4.1-Flash | code unclear | not stated | verdict and linesNo explicit no_rope_layers/nope_layer_interval config key or code path is visible in the given files to determine whether some softmax attention layers skip positional encoding entirely. | code → |
| Hy4-preview | not in its code | not stated | verdict and linesEvery attention layer applies rotary embeddings to its rope slice uniformly (via the shared HYV4RotaryEmbedding and apply_rotary_pos_emb call in every HYV4Attention forward); there is no config or code path that skips RoPE entirely for a subset of softmax attention layers. | code → |
| GLM-5.3-Flash | not in its code | not stated | mla_use_nope=true · qk_rope_head_dim=0All softmax-attention (indexed/DSA) layers uniformly use no positional encoding (qk_rope_head_dim=0, mla_use_nope=true, position_embeddings=None for every layer) — this is a uniform NoPE setup rather than some layers having RoPE and others not. text_config.mla_use_nope = truetext_config.qk_rope_head_dim = 0 | code → |
| Qwen3.8-Flash-Next | not in its code | not stated | verdict and linesAll softmax-attention (indexed_attention/full_attention) layers apply the same partial RoPE via apply_rotary_pos_emb; there is no per-layer toggle disabling RoPE entirely on some attention layers (linear_attention layers are not softmax attention and don't count). | code → |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | verdict and linesEvery attention layer applies rotary embedding (either main or compress rope type) on its trailing rope slice; there is no layer that skips positional encoding entirely, only partial RoPE per head. | code → |
| GLM-5.3 | not in its code | not stated | verdict and linesEvery attention layer is the same GlmMoeDsaAttention applying rotary embeddings to the rope slice; there is no per-layer toggle disabling RoPE entirely for some softmax attention layers. | code → |
| DeepSeek-V4-Pro-0813 | not in its code | not stated | verdict and linesEvery attention layer applies rotary position embedding (either main theta=10000 or compress theta=160000 with yarn) to its rope slice; there is no layer type that skips positional encoding entirely | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | code unclear | adopted | verdict and linesThe attention module's forward does not show rotary embedding application at all (no cos/sin passed into NemotronHAttention.forward), so it cannot be determined whether some full-attention layers skip positional encoding while others apply it. | code → |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | verdict and linesAll attention layers apply partial RoPE to the trailing rope slice of every head; none of the softmax attention layers skip positional encoding entirely, so there is no NoPE/RoPE layer split. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesEvery Qwen3VLTextDecoderLayer uses the same attention class with RoPE applied uniformly; there is no per-layer switch omitting positional encoding. | code → |
| Laguna-S-2.1 | not in its code | not stated | rope_type=defaultEvery layer type (full_attention and sliding_attention) has a rope_parameters entry with a defined rope_type, so no softmax attention layer skips positional encoding entirely. rope_parameters.sliding_attention.rope_type = "default" | code → |
| Kimi K3 | not in its code | adopted | mla_use_nope=trueAll full-attention (MLA) layers use mla_use_nope=True uniformly, meaning nope is applied to every softmax-attention layer's designated portion rather than some layers having positional encoding and others not; there is no layer-varying NoPE pattern. text_config.mla_use_nope = true | code → |
| Inkling | code unclear | not stated | verdict and linesThe model has no RoPE at all (it uses a learned relative-position bias instead), so there is no distinction between layers with and without positional encoding visible in this code; this check does not clearly apply and the files don't show any per-layer toggle for positional encoding. | code → |
| Hy3 | not in its code | not stated | verdict and linesRoPE (via position_embeddings) is applied uniformly to every attention layer with no per-layer no-rope switch. | code → |
| GLM-5.2 | not in its code | not stated | verdict and linesEvery attention layer applies the same decoupled RoPE scheme (rotary applied to the rope slice); there is no subset of softmax attention layers lacking positional encoding entirely. | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | code unclear | not stated | verdict and linesNemotronHAttention never calls apply_rotary_pos_emb or references rotary embeddings in its forward, and no rotary embedding module is instantiated in the given files, so whether full_attention layers receive any positional encoding cannot be determined from this code. | code → |
| MiniMax-M3 | not in its code | not stated | verdict and linesAll attention layers apply the same rotary embedding via apply_rotary_pos_emb; there is no per-layer toggle to skip positional encoding entirely. | code → |
| Step-3.7-Flash | not in its code | not stated | use_rope_layers=[]Every layer_type (full_attention and sliding_attention) gets an entry in rope_parameters with rope_type 'default' plus rope_theta, so all attention layers receive positional encoding; there is no no_rope_layers mechanism in the code. text_config.use_rope_layers = [] | code → |
| MolmoAct 2 | not in its code | not stated | rope_scaling_layers=nullEvery decoder layer calls apply_rotary_pos_emb on position_embeddings unconditionally; rope_scaling_layers is null so there's no per-layer selection skipping RoPE on some attention layers. text_config.rope_scaling_layers = null | code → |
| DeepSeek-V4-Flash | not in its code | not stated | qk_rope_head_dim=64All attention layers apply partial RoPE to the same trailing slice of each head (the leading slice is always unrotated, not a whole layer being without any positional encoding), so there is no layer-level NoPE/RoPE split across whole softmax-attention layers. qk_rope_head_dim = 64 | code → |
| DeepSeek-V4-Pro | not in its code | not stated | verdict and linesAll attention layers apply partial RoPE (either 'main' or 'compress' rope type) to their rope slice; there is no attention layer that skips positional encoding entirely. | code → |
| MiMo-V2.5 | not in its code | not stated | verdict and linesAll softmax attention layers apply RoPE to the rope_dim portion of every head (partial RoPE uniformly applied); there is no per-layer toggle disabling RoPE entirely on some attention layers. | code → |
| MiMo-V2.5-Pro | not in its code | not stated | verdict and linesAll softmax-attention layers apply the same partial_rotary_factor RoPE scheme uniformly (full-attention and sliding-window layers both get rotary embeddings via rotary_emb/swa_rotary_emb); there is no layer subset entirely without positional encoding. | code → |
| Qwen3.6-35B-A3B | not in its code | not stated | verdict and linesAll full_attention layers apply rotary embeddings uniformly via apply_rotary_pos_emb; there is no per-layer flag disabling RoPE for some softmax attention layers (linear-attention layers lacking RoPE don't count). | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesEvery unique layer_type (sliding_attention, full_attention) has an associated rope_parameters entry and the rotary embedding is applied unconditionally per layer type, so no attention layer lacks positional encoding. | code → |
| Qwen3.5-397B-A17B | not in its code | not stated | verdict and linesAll full_attention layers apply rotary embeddings via position_embeddings/apply_rotary_pos_emb; there is no no_rope_layers config or per-layer skip of RoPE among attention layers. | code → |
| DeepSeek-V3.2 | not in its code | not stated | verdict and lineslayer_types is uniformly 'indexed_attention' for all layers and every layer applies the same partial RoPE pattern via apply_rotary_pos_emb_interleave; no layer fully skips positional encoding. | code → |
| X-VLA | not in its code | not stated | verdict and linesAll attention layers in the text model use the same learned absolute positional embedding scheme uniformly; there is no mixture of layers with and without positional encoding. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesAll attention layers receive the same rotary position embeddings computed once and passed to every layer; there is no layer-conditional skip of RoPE | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo no_rope_layers, nope_layer_interval, or layer_types config exists, and no code path skips positional encoding for some attention layers. | code → |
4 quotes from 3 sources, one card per model, each opening with its strongest.
the full-attention blocks use No Positional Encoding (NoPE)
Nemotron 3 models do not use RoPE in attention layers and therefore do not suffer from out-of-distribution RoPE issues during context extension
Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA.
Kimi K3 follows the hybrid design of Kimi Linear [63] and applies No Position Encoding (NoPE) to all MLA layers.