gpt-oss-120b accepts 131,072 tokens (max_position_embeddings: 131072), and its config records
the shorter length it started from (initial_context_length: 4096; the model card gives no
pre-training length). The card says how it got there: “We apply rotary position embeddings and
extend the context length of dense layers to 131,072 tokens using YaRN”. In the config that is
one block of six fields, rope_scaling: {factor: 32.0, original_max_position_embeddings: 4096, beta_fast: 32.0, beta_slow: 1.0, rope_type: "yarn", truncate: false}. Without it, in the 18
layers that attend to the whole context, the slowest rotating pair of every head would be asked
to read an angle 32 times larger than any it met at 4,096 tokens.
Nothing in attention’s dot products knows which token came first. Almost every model in this map adds order with rotary position embedding (RoPE), which rotates queries and keys by an angle that grows with position. RoPE ties its slowest pairs to the lengths seen in training, so a longer window needs rescaling, more training, or layers that carry no position at all.
Background
Attention without position is invariant to the order of its keys: shuffle keys and values together and each output is unchanged. A decoder’s causal mask, which lets token attend only to tokens , still leaks some order (see NoPE).
A global (full) layer lets each token attend to every earlier token; a local or sliding-window layer only to the previous , so the offsets it sees stay below at any context length (T2). During generation a model keeps past keys and values in a KV cache (T1) and adds one token per decode step.
The original transformer added a sinusoidal or learned position vector to each embedding. A learned table ends at its last row, and the score of against has cross terms such as that mix content with absolute positions.
Notation: head dimension, and a query’s and a key’s position, , the length trained before extension, the target, , the dimensions that carry RoPE (as in T1), and “base” for RoPE’s constant ( counts bytes here).
Rotary position embedding
The mechanism
RoPE (Su et al., RoFormer, 2021) splits a head’s query and key vectors into pairs of coordinates and treats each pair as a point in a plane. At position , pair of the query is rotated by the angle , and pair of a key at position by , with
the ladder of frequencies of the original sinusoidal embedding, applied by rotation instead of addition. A rotation preserves lengths, and the dot product of two vectors rotated in the same plane depends only on the difference of their angles. For one pair, with the rotation by ,
so the score depends on the contents and on the offset, never on or alone (RoFormer eq. 16). Summed over pairs, it is a sum of sinusoids in the offset, one per pair, whose amplitudes and phases the contents set:
No parameters are learned, and keys enter the KV cache already rotated. Pair 0 always turns by 1 radian per token and comes back to its start every tokens. Each later pair turns slower by a factor , and the last by radians per token, which is times (1.45 times for gpt-oss). One full turn takes tokens, the pair’s wavelength, and a pair can’t tell from : fast pairs resolve small differences and repeat every few tokens, slow pairs are coarse but don’t repeat for a long way.
- pairs showing an angle never seen in training (outlined)
- 12 of 32
- pairs that never finish a turn in 4,096 positions (wedge, not disc)
- 14 of 32
- slowest pair (pair 31), angle unwrapped
- 1.27 rad, against 0.0396 rad in training (×32)
angles seen in training (Δ = 0 to 4,096)hand at the chosen offsetoutside its trained range18 bold number: never finishes a turn in 4,096 positions
What goes wrong past the trained length
For gpt-oss (rope_theta: 150000, head_dim: 64, so 32 pairs), pair 0 completes about 652 turns
over 4,096 positions and pair 31, with a wavelength of about 649,000 tokens, completes 0.006 of a
turn (computed). Fourteen of the 32 pairs, numbers 18 to 31, never finish a turn during the 4,096
positions: training showed them only part of the circle, from 80% of a turn for pair 18 to 0.6%
for pair 31. Such a pair never repeats an angle in training, so its angle names the offset
uniquely and can serve as a plain distance gauge. YaRN’s authors argue the network uses it as one:
in dimensions whose wavelength exceeds the training length “we presume having unique position
pairs implies that the absolute positional information remains intact” (Peng et al., §3.2). Their
“absolute” is the distance to the first token; the score still depends on only. At offset 131,072 pair 31
sits at 1.27 rad, 32 times its largest trained angle, a value the weights reading it were never
fitted to. Extension methods differ in how they treat these slow pairs.
Raising the base
Raising the base, say from 10,000 to 500,000, slows every pair except pair 0, so a longer stretch of positions fits into the arc each slow pair covered in training. Xiong et al. (2023) called this adjusted base frequency (ABF) and extended Llama 2 to 32,768 tokens with it and continued pre-training. Llama 3 used 500,000 from the start.
Their stated reason was a decay, rather than unseen angles: the default base “imposes a heavy decay on the attention scores”. RoFormer proves that an upper bound on the score shrinks with distance; the score itself need not, and for independent Gaussian queries and keys its expectation is zero at every distance (Barbero et al., 2024, Prop. 3.2). What does fall off is the score of a query and key that point the same way: each pair contributes , near 1 until the pair has turned and near 0 on average after, so the score follows the share of pairs slower than , roughly (our approximation). Over 64 pairs at 1,000 tokens the mean cosine is 0.16 at base 10,000 and 0.49 at 500,000 (computed; the approximation gives 0.25 and 0.47).
A base change alters what every slow pair means, so it comes with training at the new length. MiMo-V2-Flash pre-trained at 32,768 tokens with base 640,000 in its global layers; then “the sequence length is extended to 262,144, and the RoPE base frequency of GA is adjusted to 5,000,000”, in a stage that ran from 26T to 27T tokens (MiMo-V2.5 and V2.6 use 1e7).
Configured bases run from 10,000, the norm in local layers, to Hy3’s 11,158,840; 5e6 to 1e7 (Qwen, MiMo, MiniMax, Hy4, GLM-5.2 and 5.3, Step-3.7, Step-5) sit in global or all layers.
A larger base is not always better for retrieval. Long-context tests mostly hide one fact (a “needle”) in filler text and ask for it; RULER (Hsieh et al., 2024) is a suite of such tasks, scored in percent. In Cohere’s RoPE/NoPE hybrid (Yang et al., 2025, below), raising the RoPE layers’ base from 10,000 to 4 million in fine-tuning lowered the needle score at 128K from 8.04 to 6.20 (best runs near 10).
Rescaling the frequencies
The next methods compute new frequencies from and instead of choosing a new base. They can be applied to a finished model at load time, which is how the papers first test them, but in the mapped models they are mostly part of the training: the lab switches the scaling on for the long-context stages and ships the checkpoint with it.
Position interpolation and NTK-aware scaling
Position interpolation (PI, Chen et al., 2023) divides every position by before the rotation, which is the same as dividing every by . All angles at the new length then fall inside the trained range. Its Theorem 2.1 bounds how far the score at a fractional offset strays from the straight line between its integer neighbours, at least about 600 times tighter than RoFormer’s bound on the score itself in LLaMA 7B’s setting. PI extended LLaMA 7B to 65B to 32,768 tokens with at most 1,000 fine-tuning steps.
PI also slows the fast pairs. With gpt-oss’s , neighbouring tokens that differed by a radian in pair 0 would differ by 0.03 (computed). Read the other way, the theorem shows the cost: the offsets 0 to 32 now all sit between the old offsets 0 and 1, where the score is close to a straight line, so a head can barely single out the previous token (our reading). YaRN’s authors trace PI’s limits to this (“it removes the high frequency components of RoPE”) and report that PI fine-tunes degraded beyond a factor of about 8 (Peng et al., §3.1).
The “NTK-aware” rule, posted to Reddit by bloc97 in 2023 and summarized in YaRN’s appendix A.2, changes the base instead: , which gives . Pair 0 stays exactly as it was and the slowest pair is divided by exactly . The name refers to Tancik et al. (2020): by neural-tangent-kernel theory, networks fit high-frequency functions only if the input embedding contains high frequencies (YaRN §3.1). For gpt-oss the base would be about 5.37 million (computed). It is ABF with the base computed from instead of chosen and trained. It beats PI without fine-tuning, but only the slowest pair lands exactly in range: for gpt-oss, pairs 18 to 30 still reach 1.1 to 4.3 times the largest angle they saw in training (computed), and after fine-tuning it came out worse than PI (YaRN, A.2).
YaRN
YaRN (Peng et al., 2023) sorts the pairs by how many turns they complete over the original length. Pairs with (fast, many turns) are left untouched. Pairs with (less than a turn) are divided by , exactly as in PI. Between them each frequency is a weighted mix,
with rising from 0 at to 1 at . The paper makes linear
in . Hugging Face’s code, which the mapped configs run, makes it linear in the pair index ,
which moves the middle pairs much less. The paper’s and are the configs’
beta_slow: 1 and beta_fast: 32, counted in turns over original_max_position_embeddings.
YaRN also sharpens the softmax, which flattens as more keys compete: it divides the scores by a temperature with , fitted on LLaMA models (eq. 15; the paper writes ). Implementations multiply the cosine and sine by , so the rotated part of each score grows by . With partial RoPE (below) the content part is left alone, so the factor shifts weight from content to position instead.
For gpt-oss, YaRN leaves pairs 0 to 8 alone, interpolates 18 to 31, and ramps 9 to 17
(computed with Hugging Face’s _compute_yarn_parameters; truncate: false leaves the band edges
unrounded). Pair 12 keeps 59% of its frequency in that ramp, 23% in the paper’s. All 64 dimensions
are rotated, so the scores grow by . The paper extended Llama 2 7B and 13B to 64K with 400 fine-tuning steps at batch size 64, and
to 128K with 200 more steps, still on 64K-token sequences (§4.1).
θ′/θ, the factor each pair's frequency is multiplied by (log scale)
- YaRN groups
- 9 untouched, 9 on the ramp, 14 interpolated
- stretch
- 4,096 → 131,072 positions; base 150,000, 32 pairs; Llama-3 factors 1, 4
Llama 3.1’s scaling, described only in code (Meta’s apply_scaling, Hugging Face’s llama3
type), is the same scheme without the temperature, with a ramp linear in and, in YaRN’s terms,
, (our reading). Step-3.7-Flash uses it in its full-attention layers, factor
2 from 131,072, with high_freq_factor: 32, YaRN’s default edges.
Trained with it, or switched on later
Three regimes hide behind the same config block. In Mistral Large 4 (and DeepSeek-V4, on
our reading below) the scaling is on during long-context training and stays on; Laguna-S-2.1’s
documents don’t say. Qwen3.5 to 3.8 ship without scaling at
262,144 tokens, and the cards offer YaRN beyond that native length (factor: 4.0 for about a million
tokens; “if the typical context length for your application is 524,288 tokens, it would be better
to set factor as 2.0”, see YaRN factor
selection). Poolside’s Laguna XS.2 report does both: YaRN in training to 128K, then 256K
“without any training by doubling the RoPE scale”.
Every code path we could read applies the scaling statically, computed once at load for every input. YaRN’s paper also describes a dynamic variant, for the current length , and warns that a fixed factor costs quality on short inputs, which see interpolated slow pairs and a sharper softmax they don’t need; Qwen’s cards give the same warning for their opt-in.
The KV cache nearly forces the static choice. A cached key is stored after its rotation, so it is valid only for the frequencies it was rotated with: change the factor and every cached prefix is wrong; change it mid-sequence, as dynamic scaling does, and old and new keys disagree (YaRN’s authors recommend caching keys before rotation). It also shapes cache formats: DeepSeek-V4 keeps the 64 rotary dimensions in BF16 and the rest in FP8, and V4.1 quantizes “the cache after RoPE”.
Who uses it
Eleven of the 35 checked language models set YaRN, from four labs (factors in the table below).
DeepSeek-V3.2’s MLA heads rotate 64 of 192 query-key dimensions, and mscale_all_dim: 1.0 moves
the factor to the softmax scale and all scores grow by (computed).
DeepSeek-V4 applies YaRN only in layers that attend over compressed blocks of keys (T3), at base
160,000, with the temperature forced to 1. No DeepSeek document in the corpus mentions YaRN, and
V4’s report trains “to 16K, 64K, and 1M”, so the 1M stage most likely ran with the scaled
frequencies (our reading). Laguna-S-2.1 applies YaRN in its 12 global layers with
attention_factor: 1.4852030263919618, , which multiplies the rotated half of
each score by 2.21 (computed).
Mistral Large 4 learns the frequencies:
LeRoPE multiplies each YaRN
frequency by a learned , starting at 1 (our reading) and trained by the language-model loss. With it comes SSMax, a learned
per-head query scale in every layer, NoPE layers included, in place of YaRN’s
fixed temperature (our reading; the config sets attention_factor: 1.0); a log-length scale keeps
the largest softmax weight from shrinking as keys accumulate. In Mistral’s model-surgery runs
(changing a trained model’s architecture and training on, at 8×10²² FLOPs), the pair “improved
downstream performance over fixed YaRN, despite nearly identical training loss” on SWE-bench
Verified, and in an intermediate checkpoint two bands “near the 4K training context” had risen
about 1.5 and 1.8 times over YaRN. The block’s “2,048 from 512”, although Mistral pre-trained at 4K
and 40K, matches the window its RoPE layers had before the surgery (our reading).
Partial RoPE
Partial RoPE rotates only of a head’s dimensions, so the score splits into a positional and a content part:
with the block-diagonal matrix of the rotations by . The content part scores a key the same at any distance. In the standard implementation (GPT-NeoX’s, Hugging Face’s default) the frequencies are computed over , so the rotated pairs span the whole range in fewer pairs and still need rescaling; what partial rotation does for length is make a smaller share of each score depend on position.
Qwen3.5, 3.6 and 3.8 rotate 64 of 256 dimensions (partial_rotary_factor: 0.25), MiMo 64 of 192
(0.334, in both its local and global layers), MiniMax-M3 64 of 128 (0.5). Laguna-S-2.1,
Step-3.7-Flash and Step-5 rotate a half or a third in their global layers and every dimension in
their local ones.
Multi-head latent attention (MLA, T1) is a forced case: in decode the key up-projection is folded into the query, and a rotation by each cached key’s position would sit between them, so each head gets a separate rotary part of with one rotary key shared by all heads (DeepSeek-V3.2, GLM-5.2 and 5.3, Hy4-preview, Mistral Large 4). DeepSeek-V4’s single key-value head serves as both keys and values and rotates the last 64 of 512 dimensions; since the values are then rotated, V4 rotates the output back by the query’s position so that it carries “relative position embeddings”.
Standard partial RoPE costs no bytes or FLOPs. MLA’s rotary key costs 64 cached elements beside
the 512-element latent, 11% of each MLA entry (computed); GLM-5.3-Flash, with no position in its MLA
layers, drops it (qk_rope_head_dim: 0), while Kimi K3 still caches it unrotated. In expressiveness,
partial RoPE costs frequencies: fewer over the same band. The one public ablation in the corpus,
Poolside’s on a 16.6B-total, 2.3B-active proxy, moved the benchmark average with 50% partial RoPE
in the global layers from 0.5328 to 0.5425 at 4K, 0.267 to 0.266 at 32K and 0.272 to 0.284 at 128K
(Laguna report, Table 9); the report treats 4K differences below about 0.005 as noise and gives no
noise level for the others.
Proportional RoPE
Barbero et al. (2024) found that a trained Gemma 7B uses its fastest pairs for positional heads (attending to the previous token, or along a diagonal) and its slowest for what they “suspect are used to carry semantic information”. Over long distances even a slow rotation misaligns such a channel (Theorem 6.1, for a single frequency), so their p-RoPE removes it from those pairs. It keeps the fastest fraction of a full head’s pairs at their usual frequencies and sets the rest to zero; is NoPE, RoPE. Unlike partial RoPE, it keeps only the fast end of the spectrum.
Gemma 4 31B’s global layers (rope_type: "proportional", partial_rotary_factor: 0.25, base
1,000,000, head dimension 512, twice its local layers’ 256) rotate 64 of 256 pairs, whose longest
wavelength is 188 tokens; a standard partial RoPE with the same 128 rotated dimensions would reach
about 5.1 million (computed). The rotated part of the score in those layers varies only on scales
of a few hundred tokens: averaged over thousands it is flat, with no trend from near to far, and
the other 384 dimensions contribute the same at every distance. The local layers keep full RoPE at base 10,000 over 1,024-token
windows.
Barbero et al. trained Gemma 2B models at 8,192 tokens and report perplexity only: beat
full RoPE slightly (4.4414 against 4.4627 on Wiki), and , Gemma 4’s value, was worse
(4.5302; single runs). Long contexts were not tested. Gemma 4’s report states the choice (“p-RoPE
with p=0.25 on global attention layers”) and one consequence, a global KV cache “effectively”
37.5% smaller. On our reading that follows from p-RoPE: the global layers reuse the key
projection as the value before rotation (attention_k_eq_v: true), so key and value can share
storage only where the key isn’t rotated, and 512 + 128 of 1,024 elements is 37.5% less
(computed). The report gives no quality reason.
Liquid’s LFM3 blog is the one lab document that gives the content-channel reason: global layers
“leave the low-frequency half unrotated, so those dimensions match on content regardless of
distance” (though LFM3’s released code rotates every dimension).
No position encoding
What the causal mask gives
Remove RoPE from a decoder and it still knows order. Kazemnejad et al. (2023) prove that the first layer can recover absolute positions from the causal mask alone (their Theorem 1). A head whose keys are all equal gives every visible token the same weight ; if its value is 1 for the beginning-of-sequence token, which is always first, and 0 for every other token, it outputs at position , and the feed-forward layer can turn that into a position code. Later layers can then compare positions (Theorem 2). On synthetic tasks with about 107M-parameter models, NoPE generalized to longer inputs better than RoPE, ALiBi (a fixed per-head score penalty linear in distance) or absolute embeddings.
In the paper’s 1B-parameter test, trained at 1,024 tokens, “NoPE and ALiBi generalize similarly to larger context sizes up to almost twice their training context size”, with ALiBi more stable beyond that, while RoPE’s perplexity exploded. NoPE has no parameter tied to a length, but its signal still reaches values below that training never showed, and its softmax spreads thinner.
Why it is weak on its own
Locality is costly without explicit position. A single NoPE head whose input carries no position can’t put almost all its weight on the previous token: if the same token appears twice in a row, the query scores both copies equally (Barbero et al., Prop. 5.2). A stack can do it, by first recovering position from the mask and then comparing positions (Kazemnejad et al., Theorems 1 and 2), but it spends layers on what RoPE’s fast pairs give a single head directly. The signal also gets finer as grows: neighbours differ by a relative , 0.02% at position 4,096 (our arithmetic; the paper doesn’t discuss resolution). In Cohere’s 8B comparison, an all-NoPE model trailed RoPE after fine-tuning, validation loss 1.58 against 1.52 (Yang et al., Table 2). And Qwen tested NoPE in the full-attention layers of its Qwen3.8-Next hybrid and kept RoPE: the two “show little difference during pretraining, but the NoPE variant exhibits a substantially higher rate of endless generation after post-training” (no rates given).
Hybrids: position in some layers, none in others
Local RoPE, global NoPE
Yang et al. (2025) interleaved RoPE and NoPE layers in an 8B model and found “a spontaneous ‘division of labor’”: “RoPE layers focus on local information aggregation and NoPE layers specialize in long-range retrieval”. Their final design gives the RoPE layers a 4,096-token window and base 10,000 and makes every fourth layer a full-attention NoPE layer. Trained to 128K and tested at 256K, it held a RULER retrieval score of 74.8 where the RoPE baseline, its base raised to 8 million, fell to 57.1 (both about 96 at 8K).
A sliding-window layer never sees an offset larger than its window, so its RoPE never extrapolates: Kolibri-1’s local layers see at most 512 tokens back, at any context length.
Fill: positionRoPE (filled width = share of dims rotated)YaRN-scaled RoPEnone
Outline: reachsliding windowfull contextrecurrent state (hatched)MoE block, no token mixing
One repeating group per model, first layer at the top. * Nemotron 3 Ultra: its first nine blocks; the other groups hold the same four Mamba-2, four MoE and one attention block in varying order.
In Kolibri-1, “Positional information is injected only in the SWA blocks using RoPE with θ =
10 000. The full-attention blocks use no positional encoding”, one block in five (layer_types, 10
of 50). In “early experiments, removing RoPE from the full-attention blocks improved
long-context performance”, without numbers. The plotted comparison (Figure 7, a 7.8B-total,
0.6B-active proxy) extends a full-attention RoPE model to 64K with YaRN and trains the hybrid at
256K “without any RoPE extension”, but the arms also differ in attention type and pre-training
length, so it compares two designs, not NoPE against RoPE.
The shipped model was trained to 256K and evaluated to a million tokens with no rescaling. On Aleph Alpha’s RULER runs of base models (report, Table 18, the report’s own protocol), Kolibri-1 scores 69.8 at 256K, 65.5 at 512K and 63.2 at 1M. Nemotron 3 Nano scores 72.1, 73.1 and 58.5, and Qwen3.5-35B-A3B, served with static YaRN at factor 4 at every length, 80.1, 72.2 and 57.5. Both lead Kolibri up to 512K and trail it at 1M; Aleph Alpha’s card recommends at most 262,144 tokens for complex tasks.
Mistral Large 4 had sliding-window RoPE layers and one full-attention NoPE layer in four; the
surgery removed the windows, and “the original NoPE/global layers remained NoPE” (18 of 72 in the
config’s no_rope_layers).
Recurrent layers carry position, attention none
A linear-attention or state-space layer (T2) mixes tokens through a state. A plain running sum would be order-blind; decay gates (a learned, input-dependent form of ALiBi’s distance penalty), the delta rule’s order-dependent updates and short convolutions make these layers order- and recency-sensitive. Three families let those layers carry all position and remove it from every softmax layer:
- Kimi K3 interleaves 69 Kimi Delta Attention (KDA) layers with 24 MLA layers and “applies No Position Encoding (NoPE) to all MLA layers”; the KDA layers “provide position-sensitive and recency-aware sequence mixing”, which “avoids modifying positional-encoding parameters when extending the context length”.
- GLM-5.3-Flash interleaves 34 KDA layers with 11 sparse MLA layers and sets
qk_rope_head_dim: 0; its code passesposition_embeddings=Noneto every layer. Z.ai’s own card doesn’t mention NoPE; the vLLM recipe calls them “NoPE sparse MLA layers”. - Nemotron 3 (Ultra; 3.5 Lightning and Super) puts a few attention layers among Mamba-2 layers: “Since Mamba layers provide implicit positional information, Nemotron 3 models do not use RoPE in attention layers”.
A learned relative bias
Thinking Machines’ Inkling has no RoPE at all. Each query produces, per head, a 16-dimensional
vector (d_rel: 16), and a learned matrix maps it to one score bias per
distance: a rank-16, query-dependent relative term in the line of Shaw et al. and Music
Transformer (both 2018). The released code limits the bias to the last 1,024 tokens in the 11 global
layers (rel_extent: 1024) and 512 in the 55 local ones, so further back a global layer has no
explicit position signal, like a NoPE layer. Above 128,000 tokens the global layers also multiply
their queries by , 1.21 at 1M (computed), sharpening the softmax as
YaRN’s temperature does. The lab says this
relative attention “performs better and
extrapolates better to longer sequences” than RoPE, without numbers. No mapped model uses ALiBi.
Training at length
Twenty-three of the 35 configs declare 1,048,576 positions (computed), and every documented recipe reaches its length by training there. Kolibri-1 pre-trained 20T tokens at 16K, mid-trained (a stage between pre-training and fine-tuning) 3.4T at 64K and added 200B at 256K; DeepSeek-V4 went 4K, 16K, 64K, 1M; MiMo-V2-Flash changed its base on the way to 256K; Kimi K3 trained at 1M with no position parameter to change.
Without RoPE, a model still has to be trained at the lengths it will serve. In Nemotron 3.5’s report, after mid-training Super at 256K, “the RULER-1M score dropped from 79.9 to 33.5”, and a recovery phase of 9 billion tokens “with the same recipe” as its 1M-context stage brought it to 81.0. How labs schedule these stages, and what data they use, belongs to the training series (S2, and the page on long-context training).
Comparison
| Method | What changes | Layers | Local order | Beyond trained length | Mapped models |
|---|---|---|---|---|---|
| RoPE | rotate pairs by | all or local | strong | slow pairs out of range | nearly all |
| Larger base (ABF) | all but pair 0 slower; trained | global or all | slightly coarser | out of range later | bases 1.5e5 to 1.1e7 |
| PI | all | all | blurred | in range to | none |
| NTK-aware | base | all | nearly kept | slow-middle pairs out by up to ~4× | none |
| YaRN, Llama-3 | slow pairs , fast kept, blend; YaRN adds temperature | global or all | kept | in range to | 11 YaRN, 2 Llama-3 type |
| LeRoPE | YaRN times learned factors | RoPE layers | kept | learned | Mistral Large 4 |
| Partial RoPE | rotate dims | all or global | fewer frequencies | rotated dims still need scaling | 12, plus 12 MLA |
| p-RoPE | keep fastest fraction | global | kept | short wavelengths only | Gemma 4 |
| NoPE | none; causal mask only | global | weak alone | no frequency out of range; still trained at length | 7 |
| Learned relative bias | per-distance score bias | all | strong within extent | content only beyond | Inkling |
Where the models in the map sit
The site checks each model’s released config and modeling code for a list of features (35
language models so far). The table is read from those files, and the counts are cleaned against
the code: the stored partial-rope verdict is wrong for Kimi K3 and LFM3, and five unclear YaRN
verdicts are absent.
| Model | RoPE base | Rotated / head dims | Scaling | NoPE softmax layers |
|---|---|---|---|---|
| Kolibri-1 | 1e4 (SWA) | 128/128 (SWA) | none | 10 of 50 (all global) |
| DeepSeek-V3.2 | 1e4 | 64/192 (MLA) | YaRN 40 from 4,096 | none |
| DeepSeek-V4, V4.1 | 1.6e5 compressed layers; 1e4 sliding-only (Flash layers 0, 1) | 64/512 | YaRN 16 from 65,536, compressed layers | none |
| Gemma 4 31B | 1e4 SWA, 1e6 global | 256/256 SWA; global p-RoPE 64 of 256 pairs | none | none |
| LFM3-1.7B | 1e4 SWA, 1e6 global | 128/128 (config says 0.5 global; code rotates all) | none | none |
| MiniMax-M3, M3.1-Flash | 5e6, 1e7 | 64/128 | none | none |
| Mistral Large 4 | 1e4, learned factors | 64/128 (MLA) | YaRN 2,048 from 512 (former window, our reading) | 18 of 72 |
| Kimi K3 | none | 0 (64-dim key slice unrotated) | none | all 24 MLA (69 KDA) |
| GLM-5.3-Flash | none | 0 | none | all 11 (34 KDA) |
| GLM-5.2, 5.3 | 8e6 | 64/256 (MLA) | none | none |
| Nemotron 3 Ultra, 3.5 Lightning, Super | none | 0 | none | all attention layers (Mamba-2 rest) |
| gpt-oss-120b | 1.5e5 | 64/64 | YaRN 32 from 4,096 | none |
| Laguna-S-2.1 | 1e4 SWA, 5e5 global | 128/128 SWA, 64/128 global | YaRN 128 from 8,192, global | none |
| Qwen3.5, 3.6, 3.8 | 1e7 | 64/256 | none (YaRN opt-in) | none |
| Step-3.7-Flash | 1e4 SWA, 5e6 global | 128/128 SWA, 64/128 global | Llama-3 type 2 from 131,072, global | none |
| Step-5-Preview | 1e4 SWA, 1e7 global | 192/192 SWA, 64/192 global | Llama-3 type, factor 1 (no change) | none |
| Hy3 | 1.1e7 | 128/128 | none | none |
| Hy4-preview | 1e7 | 64/256 (MLA) | none | none |
| Inkling | no RoPE: learned relative bias | 0 | none | n/a |
| MiMo-V2.5, V2.6 | 1e4 SWA, 1e7 global | 64/192 both | none | none |
The code-check counts, and how they were cleaned:
partial-ropeis present for 25 of 35 models and 10 of 16 labs after cleaning (raw: 27). Twelve are MLA-style shared rotary keys (DeepSeek, Mistral, Tencent, Z.ai), twelve rotate a fraction of every head (Qwen, Xiaomi, MiniMax, Poolside, StepFun), and one is p-RoPE (Google). Without the MLA-style ones, 13 models from 6 labs chose partial rotation for its own sake.yarnis present for 11 models from 4 labs: DeepSeek (8), OpenAI, Poolside, Mistral; the nine with readable rotary code apply it statically.nope-layersis present for 2 models (Kolibri-1, Mistral Large 4): the check asks for some softmax layers with RoPE and some without, which excludes Kimi K3, GLM-5.3-Flash and the three Nemotron models. Counted together, 7 models from 5 labs have softmax layers without position.- Fourteen models set RoPE per layer type. gpt-oss is not one: its sliding and full layers share one RoPE, YaRN included.
Easy to get wrong
- “Partial RoPE 0.25” means two different spectra. In Qwen3.5 the factor sets and the
64 rotated dimensions span wavelengths up to 38 million tokens. In Gemma 4’s global layers the
same
partial_rotary_factor: 0.25withrope_type: "proportional"keeps only wavelengths under 188 tokens. Read therope_type, not the factor. - A config value is not a code path: LFM3’s config sets
partial_rotary_factor: 0.5for its global layers, and its released code rotates every dimension. - Turning on a model’s optional YaRN changes every key it caches. A prefix cache, or an offloaded
KV cache, built with one
rope_parameterscan’t be reused with another. - YaRN’s factor goes on the cosine and sine, scaling only rotated dimensions (all in gpt-oss, half
in Laguna’s global layers), on the softmax scale (DeepSeek-V3.2’s
mscale_all_dim), or nowhere (DeepSeek-V4, Mistral Large 4). - Which coordinates form a pair is a convention: neighbours in DeepSeek, GLM-5.2 and 5.3 and Mistral Large 4, halves elsewhere. Converting a checkpoint between them means permuting projection weights, and a mismatch fails silently.
Check yourself
- gpt-oss has base 150,000, 64 rotary dimensions and . Compute the wavelength of
pair 12 and the number of turns it completes over . What does YaRN do to it?
Answer
rad per token, so tokens and turns. That lies between and , so pair 12 is in the ramp. With the paper’s ramp, and ; Hugging Face’s ramp, linear in the pair index, keeps 0.59. - Kolibri-1’s local layers use RoPE at base 10,000 and are trained at 256K. Why don’t they need
rescaling at 1M?
Answer
They attend only to the 512 preceding tokens, so the offset in their rotation never exceeds 512, whatever the context length. Every angle they meet at 1M they met in training. - PI and NTK-aware scaling both bring the slowest pair back into range for . Why
does PI need fine-tuning while NTK-aware scaling often works without?
Answer
PI slows the fastest pairs 32-fold too, so adjacent tokens differ by 0.03 rad in pair 0 instead of a radian, a local order the model never saw. NTK-aware scaling keeps pair 0 and leaves pairs 18 to 30 partly out of range instead, which is why fine-tuning with it did worse. - Nemotron 3.5 Super has no RoPE in any layer. Why did its RULER score at 1M still fall after a
stage at 256K?
Answer
No frequency went out of range, but behaviour at 1M is learned from data at that length; after a stage at 256K it fell (79.9 to 33.5), and 9 billion tokens of long training restored it (81.0).
In the map
Methods
- Rotary Position Embedding 6 sources
- Partial rotary position embeddings 3 sources
- Proportional Rotary Position Embedding 2 sources
- YaRN 11 sources
- YaRN scaling-factor selection 2 sources
- Static YaRN 6 sources
- LeRoPE 2 sources
- No Position Encoding 3 sources
- Learned relative attention 2 sources
Taxonomy nodes
- positional encoding 24 methods
Further reading
- Su et al., RoFormer (2021), arXiv:2104.09864: the rotation’s complex-number derivation and the decay bound.
- Chen et al., Position Interpolation (2023), arXiv:2306.15595: the interpolation bound and recipe.
- Xiong et al., Effective Long-Context Scaling of Foundation Models (2023), arXiv:2309.16039: the base-frequency change and the 80B-token ablation against PI.
- Peng et al., YaRN (2023), arXiv:2309.00071: NTK-aware, NTK-by-parts, the temperature fit, dynamic scaling.
- Kazemnejad et al., The Impact of Positional Encoding on Length Generalization (2023), arXiv:2305.19466: how a causal decoder recovers position.
- Barbero et al., Round and Round We Go! (2024), arXiv:2410.06205: which frequencies Gemma uses for what, and p-RoPE.
- Yang et al., Rope to Nope and Back Again (2025), arXiv:2501.18795: the division of labour Kolibri’s design follows.
- Kolibri-1 technical report §2.1.3, §2.4.3.1, and Kimi K3 technical report §2.1.2, §3.4: the two most detailed lab accounts of a NoPE hybrid.
