Open Models Almanac

Token mixer · T4

Position encoding and length extension

Rotary position gives attention order and locality, but its slow pairs go out of range past the trained length. Rescaling and training recover length; layers without position work beside layers that carry it

Illustration of a row of round dials of increasing size mounted above a long paper tape. The small dials have hands at varied angles; the largest dial's hand has barely left its starting point. The tape runs past the last dial into open space, where one stretch of it is lightly highlighted.

gpt-oss-120b accepts 131,072 tokens (max_position_embeddings: 131072), and its config records the shorter length it started from (initial_context_length: 4096; the model card gives no pre-training length). The card says how it got there: “We apply rotary position embeddings and extend the context length of dense layers to 131,072 tokens using YaRN”. In the config that is one block of six fields, rope_scaling: {factor: 32.0, original_max_position_embeddings: 4096, beta_fast: 32.0, beta_slow: 1.0, rope_type: "yarn", truncate: false}. Without it, in the 18 layers that attend to the whole context, the slowest rotating pair of every head would be asked to read an angle 32 times larger than any it met at 4,096 tokens.

Nothing in attention’s dot products knows which token came first. Almost every model in this map adds order with rotary position embedding (RoPE), which rotates queries and keys by an angle that grows with position. RoPE ties its slowest pairs to the lengths seen in training, so a longer window needs rescaling, more training, or layers that carry no position at all.

Background

Attention without position is invariant to the order of its keys: shuffle keys and values together and each output is unchanged. A decoder’s causal mask, which lets token tt attend only to tokens 1,…,t1, \dots, t, still leaks some order (see NoPE).

A global (full) layer lets each token attend to every earlier token; a local or sliding-window layer only to the previous WW, so the offsets it sees stay below WW at any context length (T2). During generation a model keeps past keys and values in a KV cache (T1) and adds one token per decode step.

The original transformer added a sinusoidal or learned position vector to each embedding. A learned table ends at its last row, and the score of xt+ptx_t + p_t against xs+psx_s + p_s has cross terms such as xt⊤Wpsx_t^\top W p_s that mix content with absolute positions.

Notation: dhd_h head dimension, tt and ss a query’s and a key’s position, Δ=t−s\Delta = t - s, T0T_0 the length trained before extension, T′T' the target, κ=T′/T0\kappa = T'/T_0, dhRd_h^R the dimensions that carry RoPE (as in T1), and “base” for RoPE’s constant (bb counts bytes here).

Rotary position embedding

The mechanism

RoPE (Su et al., RoFormer, 2021) splits a head’s query and key vectors into dhR/2d_h^R/2 pairs of coordinates and treats each pair as a point in a plane. At position tt, pair ii of the query is rotated by the angle t θit\,\theta_i, and pair ii of a key at position ss by s θis\,\theta_i, with

θi=base−2i/dhR,i=0,…,dhR/2−1,\theta_i = \text{base}^{-2i/d_h^R}, \qquad i = 0, \dots, d_h^R/2 - 1 ,

the ladder of frequencies of the original sinusoidal embedding, applied by rotation instead of addition. A rotation preserves lengths, and the dot product of two vectors rotated in the same plane depends only on the difference of their angles. For one pair, with R(ϕ)R(\phi) the 2×22 \times 2 rotation by ϕ\phi,

(R(tθi) qi)⊤(R(sθi) ki)=qi⊤R(−Δ θi) ki,\big(R(t\theta_i)\,q_i\big)^\top \big(R(s\theta_i)\,k_i\big) = q_i^\top R\big(-\Delta\,\theta_i\big)\,k_i ,

so the score depends on the contents and on the offset, never on tt or ss alone (RoFormer eq. 16). Summed over pairs, it is a sum of sinusoids in the offset, one per pair, whose amplitudes and phases the contents set:

score(Δ)=∑i[(qi ⁣⋅ ⁣ki)cos⁡(Δθi)+(qi,1ki,2−qi,2ki,1)sin⁡(Δθi)].\text{score}(\Delta) = \sum_i \Big[(q_i\!\cdot\! k_i)\cos(\Delta\theta_i) + (q_{i,1}k_{i,2} - q_{i,2}k_{i,1})\sin(\Delta\theta_i)\Big].

No parameters are learned, and keys enter the KV cache already rotated. Pair 0 always turns by 1 radian per token and comes back to its start every 2π≈6.32\pi \approx 6.3 tokens. Each later pair turns slower by a factor base2/dhR\text{base}^{2/d_h^R}, and the last by base−(dhR−2)/dhR\text{base}^{-(d_h^R-2)/d_h^R} radians per token, which is base2/dhR\text{base}^{2/d_h^R} times 1/base1/\text{base} (1.45 times for gpt-oss). One full turn takes λi=2π/θi\lambda_i = 2\pi/\theta_i tokens, the pair’s wavelength, and a pair can’t tell Δ\Delta from Δ+λi\Delta + \lambda_i: fast pairs resolve small differences and repeat every few tokens, slow pairs are coarse but don’t repeat for a long way.

base
pairs showing an angle never seen in training (outlined)
12 of 32
pairs that never finish a turn in 4,096 positions (wedge, not disc)
14 of 32
slowest pair (pair 31), angle unwrapped
1.27 rad, against 0.0396 rad in training (×32)

angles seen in training (Δ = 0 to 4,096)hand at the chosen offsetoutside its trained range18 bold number: never finishes a turn in 4,096 positions

Figure 1.The 32 rotary pairs of one gpt-oss head. Fast pairs (top left) turned hundreds of times within the 4,096 training positions, so every angle occurred in training. The 14 slowest pairs never complete a turn in training; at an offset of 131,072, 12 of them show an angle they never saw, and the slowest sits 32 times past its trained range.

What goes wrong past the trained length

For gpt-oss (rope_theta: 150000, head_dim: 64, so 32 pairs), pair 0 completes about 652 turns over 4,096 positions and pair 31, with a wavelength of about 649,000 tokens, completes 0.006 of a turn (computed). Fourteen of the 32 pairs, numbers 18 to 31, never finish a turn during the 4,096 positions: training showed them only part of the circle, from 80% of a turn for pair 18 to 0.6% for pair 31. Such a pair never repeats an angle in training, so its angle names the offset uniquely and can serve as a plain distance gauge. YaRN’s authors argue the network uses it as one: in dimensions whose wavelength exceeds the training length “we presume having unique position pairs implies that the absolute positional information remains intact” (Peng et al., §3.2). Their “absolute” is the distance to the first token; the score still depends on Δ\Delta only. At offset 131,072 pair 31 sits at 1.27 rad, 32 times its largest trained angle, a value the weights reading it were never fitted to. Extension methods differ in how they treat these slow pairs.

Raising the base

Raising the base, say from 10,000 to 500,000, slows every pair except pair 0, so a longer stretch of positions fits into the arc each slow pair covered in training. Xiong et al. (2023) called this adjusted base frequency (ABF) and extended Llama 2 to 32,768 tokens with it and continued pre-training. Llama 3 used 500,000 from the start.

Their stated reason was a decay, rather than unseen angles: the default base “imposes a heavy decay on the attention scores”. RoFormer proves that an upper bound on the score shrinks with distance; the score itself need not, and for independent Gaussian queries and keys its expectation is zero at every distance (Barbero et al., 2024, Prop. 3.2). What does fall off is the score of a query and key that point the same way: each pair contributes cos⁡(Δθi)\cos(\Delta\theta_i), near 1 until the pair has turned and near 0 on average after, so the score follows the share of pairs slower than 1/Δ1/\Delta, roughly 1−ln⁡Δ/ln⁡base1 - \ln\Delta/\ln\text{base} (our approximation). Over 64 pairs at 1,000 tokens the mean cosine is 0.16 at base 10,000 and 0.49 at 500,000 (computed; the approximation gives 0.25 and 0.47).

A base change alters what every slow pair means, so it comes with training at the new length. MiMo-V2-Flash pre-trained at 32,768 tokens with base 640,000 in its global layers; then “the sequence length is extended to 262,144, and the RoPE base frequency of GA is adjusted to 5,000,000”, in a stage that ran from 26T to 27T tokens (MiMo-V2.5 and V2.6 use 1e7).

Configured bases run from 10,000, the norm in local layers, to Hy3’s 11,158,840; 5e6 to 1e7 (Qwen, MiMo, MiniMax, Hy4, GLM-5.2 and 5.3, Step-3.7, Step-5) sit in global or all layers.

A larger base is not always better for retrieval. Long-context tests mostly hide one fact (a “needle”) in filler text and ask for it; RULER (Hsieh et al., 2024) is a suite of such tasks, scored in percent. In Cohere’s RoPE/NoPE hybrid (Yang et al., 2025, below), raising the RoPE layers’ base from 10,000 to 4 million in fine-tuning lowered the needle score at 128K from 8.04 to 6.20 (best runs near 10).

Rescaling the frequencies

The next methods compute new frequencies from T0T_0 and κ\kappa instead of choosing a new base. They can be applied to a finished model at load time, which is how the papers first test them, but in the mapped models they are mostly part of the training: the lab switches the scaling on for the long-context stages and ships the checkpoint with it.

Position interpolation and NTK-aware scaling

Position interpolation (PI, Chen et al., 2023) divides every position by κ\kappa before the rotation, which is the same as dividing every θi\theta_i by κ\kappa. All angles at the new length then fall inside the trained range. Its Theorem 2.1 bounds how far the score at a fractional offset strays from the straight line between its integer neighbours, at least about 600 times tighter than RoFormer’s bound on the score itself in LLaMA 7B’s setting. PI extended LLaMA 7B to 65B to 32,768 tokens with at most 1,000 fine-tuning steps.

PI also slows the fast pairs. With gpt-oss’s κ=32\kappa = 32, neighbouring tokens that differed by a radian in pair 0 would differ by 0.03 (computed). Read the other way, the theorem shows the cost: the offsets 0 to 32 now all sit between the old offsets 0 and 1, where the score is close to a straight line, so a head can barely single out the previous token (our reading). YaRN’s authors trace PI’s limits to this (“it removes the high frequency components of RoPE”) and report that PI fine-tunes degraded beyond a factor of about 8 (Peng et al., §3.1).

The “NTK-aware” rule, posted to Reddit by bloc97 in 2023 and summarized in YaRN’s appendix A.2, changes the base instead: base′=base⋅κdhR/(dhR−2)\text{base}' = \text{base} \cdot \kappa^{d_h^R/(d_h^R-2)}, which gives θi′=θi κ−2i/(dhR−2)\theta_i' = \theta_i\,\kappa^{-2i/(d_h^R-2)}. Pair 0 stays exactly as it was and the slowest pair is divided by exactly κ\kappa. The name refers to Tancik et al. (2020): by neural-tangent-kernel theory, networks fit high-frequency functions only if the input embedding contains high frequencies (YaRN §3.1). For gpt-oss the base would be about 5.37 million (computed). It is ABF with the base computed from κ\kappa instead of chosen and trained. It beats PI without fine-tuning, but only the slowest pair lands exactly in range: for gpt-oss, pairs 18 to 30 still reach 1.1 to 4.3 times the largest angle they saw in training (computed), and after fine-tuning it came out worse than PI (YaRN, A.2).

YaRN

YaRN (Peng et al., 2023) sorts the pairs by how many turns ri=T0/λir_i = T_0/\lambda_i they complete over the original length. Pairs with ri>βr_i > \beta (fast, many turns) are left untouched. Pairs with ri<αr_i < \alpha (less than a turn) are divided by κ\kappa, exactly as in PI. Between them each frequency is a weighted mix,

θi′=(1−γi) θiκ+γi θi,\theta_i' = (1-\gamma_i)\,\frac{\theta_i}{\kappa} + \gamma_i\,\theta_i ,

with γi\gamma_i rising from 0 at ri=αr_i = \alpha to 1 at ri=βr_i = \beta. The paper makes γ\gamma linear in rir_i. Hugging Face’s code, which the mapped configs run, makes it linear in the pair index ii, which moves the middle pairs much less. The paper’s α=1\alpha = 1 and β=32\beta = 32 are the configs’ beta_slow: 1 and beta_fast: 32, counted in turns over original_max_position_embeddings.

YaRN also sharpens the softmax, which flattens as more keys compete: it divides the scores by a temperature τ<1\tau < 1 with 1/τ=0.1ln⁡κ+1\sqrt{1/\tau} = 0.1 \ln \kappa + 1, fitted on LLaMA models (eq. 15; the paper writes tt). Implementations multiply the cosine and sine by 1/τ\sqrt{1/\tau}, so the rotated part of each score grows by 1/τ=(0.1ln⁡κ+1)21/\tau = (0.1\ln\kappa + 1)^2. With partial RoPE (below) the content part is left alone, so the factor shifts weight from content to position instead.

For gpt-oss, YaRN leaves pairs 0 to 8 alone, interpolates 18 to 31, and ramps 9 to 17 (computed with Hugging Face’s _compute_yarn_parameters; truncate: false leaves the band edges unrounded). Pair 12 keeps 59% of its frequency in that ramp, 23% in the paper’s. All 64 dimensions are rotated, so the scores grow by 1.3472=1.811.347^2 = 1.81. The paper extended Llama 2 7B and 13B to 64K with 400 fine-tuning steps at batch size 64, and to 128K with 200 more steps, still on 64K-token sequences (§4.1).

θ′/θ, the factor each pair's frequency is multiplied by (log scale)

YaRN groups
9 untouched, 9 on the ramp, 14 interpolated
stretch
4,096 → 131,072 positions; base 150,000, 32 pairs; Llama-3 factors 1, 4
Figure 2.How the rescaling rules change each rotary pair of gpt-oss when stretching 4,096 positions to 131,072. Position interpolation slows every pair 32-fold, including the fast ones that encode nearby order. YaRN leaves the nine fastest pairs alone and interpolates only the 14 that never finished a turn in training. Inside the ramp, Hugging Face's code (solid) and the YaRN paper (dashed) differ: pair 12 keeps 59% of its frequency in one and 23% in the other.

Llama 3.1’s scaling, described only in code (Meta’s apply_scaling, Hugging Face’s llama3 type), is the same scheme without the temperature, with a ramp linear in rr and, in YaRN’s terms, α=1\alpha = 1, β=4\beta = 4 (our reading). Step-3.7-Flash uses it in its full-attention layers, factor 2 from 131,072, with high_freq_factor: 32, YaRN’s default edges.

Trained with it, or switched on later

Three regimes hide behind the same config block. In Mistral Large 4 (and DeepSeek-V4, on our reading below) the scaling is on during long-context training and stays on; Laguna-S-2.1’s documents don’t say. Qwen3.5 to 3.8 ship without scaling at 262,144 tokens, and the cards offer YaRN beyond that native length (factor: 4.0 for about a million tokens; “if the typical context length for your application is 524,288 tokens, it would be better to set factor as 2.0”, see YaRN factor selection). Poolside’s Laguna XS.2 report does both: YaRN in training to 128K, then 256K “without any training by doubling the RoPE scale”.

Every code path we could read applies the scaling statically, computed once at load for every input. YaRN’s paper also describes a dynamic variant, κ=max⁡(1,l/T0)\kappa = \max(1, l/T_0) for the current length ll, and warns that a fixed factor costs quality on short inputs, which see interpolated slow pairs and a sharper softmax they don’t need; Qwen’s cards give the same warning for their opt-in.

The KV cache nearly forces the static choice. A cached key is stored after its rotation, so it is valid only for the frequencies it was rotated with: change the factor and every cached prefix is wrong; change it mid-sequence, as dynamic scaling does, and old and new keys disagree (YaRN’s authors recommend caching keys before rotation). It also shapes cache formats: DeepSeek-V4 keeps the 64 rotary dimensions in BF16 and the rest in FP8, and V4.1 quantizes “the cache after RoPE”.

Who uses it

Eleven of the 35 checked language models set YaRN, from four labs (factors in the table below). DeepSeek-V3.2’s MLA heads rotate 64 of 192 query-key dimensions, and mscale_all_dim: 1.0 moves the factor to the softmax scale and all scores grow by (0.1ln⁡40+1)2=1.87(0.1\ln 40 + 1)^2 = 1.87 (computed). DeepSeek-V4 applies YaRN only in layers that attend over compressed blocks of keys (T3), at base 160,000, with the temperature forced to 1. No DeepSeek document in the corpus mentions YaRN, and V4’s report trains “to 16K, 64K, and 1M”, so the 1M stage most likely ran with the scaled frequencies (our reading). Laguna-S-2.1 applies YaRN in its 12 global layers with attention_factor: 1.4852030263919618, 0.1ln⁡128+10.1\ln 128 + 1, which multiplies the rotated half of each score by 2.21 (computed).

Mistral Large 4 learns the frequencies: LeRoPE multiplies each YaRN frequency by a learned eαie^{\alpha_i}, starting at 1 (our reading) and trained by the language-model loss. With it comes SSMax, a learned per-head query scale alog⁡(t+1)+ba\log(t+1) + b in every layer, NoPE layers included, in place of YaRN’s fixed temperature (our reading; the config sets attention_factor: 1.0); a log-length scale keeps the largest softmax weight from shrinking as keys accumulate. In Mistral’s model-surgery runs (changing a trained model’s architecture and training on, at 8×10²² FLOPs), the pair “improved downstream performance over fixed YaRN, despite nearly identical training loss” on SWE-bench Verified, and in an intermediate checkpoint two bands “near the 4K training context” had risen about 1.5 and 1.8 times over YaRN. The block’s “2,048 from 512”, although Mistral pre-trained at 4K and 40K, matches the window its RoPE layers had before the surgery (our reading).

Partial RoPE

Partial RoPE rotates only dhRd_h^R of a head’s dhd_h dimensions, so the score splits into a positional and a content part:

(Rtqt)⊤(Rsks)=qtR⊤RΘ(−Δ) ksR+qtC⊤ksC,(R_t q_t)^\top (R_s k_s) = q_t^{R\top} R_\Theta(-\Delta)\, k_s^R + q_t^{C\top} k_s^C ,

with RΘ(ϕ)R_\Theta(\phi) the block-diagonal matrix of the dhR/2d_h^R/2 rotations by ϕθi\phi\theta_i. The content part scores a key the same at any distance. In the standard implementation (GPT-NeoX’s, Hugging Face’s default) the frequencies are computed over dhRd_h^R, so the rotated pairs span the whole range in fewer pairs and still need rescaling; what partial rotation does for length is make a smaller share of each score depend on position.

Qwen3.5, 3.6 and 3.8 rotate 64 of 256 dimensions (partial_rotary_factor: 0.25), MiMo 64 of 192 (0.334, in both its local and global layers), MiniMax-M3 64 of 128 (0.5). Laguna-S-2.1, Step-3.7-Flash and Step-5 rotate a half or a third in their global layers and every dimension in their local ones.

Multi-head latent attention (MLA, T1) is a forced case: in decode the key up-projection is folded into the query, and a rotation by each cached key’s position would sit between them, so each head gets a separate rotary part of dhR=64d_h^R = 64 with one rotary key shared by all heads (DeepSeek-V3.2, GLM-5.2 and 5.3, Hy4-preview, Mistral Large 4). DeepSeek-V4’s single key-value head serves as both keys and values and rotates the last 64 of 512 dimensions; since the values are then rotated, V4 rotates the output back by the query’s position so that it carries “relative position embeddings”.

Standard partial RoPE costs no bytes or FLOPs. MLA’s rotary key costs 64 cached elements beside the 512-element latent, 11% of each MLA entry (computed); GLM-5.3-Flash, with no position in its MLA layers, drops it (qk_rope_head_dim: 0), while Kimi K3 still caches it unrotated. In expressiveness, partial RoPE costs frequencies: fewer over the same band. The one public ablation in the corpus, Poolside’s on a 16.6B-total, 2.3B-active proxy, moved the benchmark average with 50% partial RoPE in the global layers from 0.5328 to 0.5425 at 4K, 0.267 to 0.266 at 32K and 0.272 to 0.284 at 128K (Laguna report, Table 9); the report treats 4K differences below about 0.005 as noise and gives no noise level for the others.

Proportional RoPE

Barbero et al. (2024) found that a trained Gemma 7B uses its fastest pairs for positional heads (attending to the previous token, or along a diagonal) and its slowest for what they “suspect are used to carry semantic information”. Over long distances even a slow rotation misaligns such a channel (Theorem 6.1, for a single frequency), so their p-RoPE removes it from those pairs. It keeps the fastest fraction pp of a full head’s pairs at their usual frequencies and sets the rest to zero; p=0p = 0 is NoPE, p=1p = 1 RoPE. Unlike partial RoPE, it keeps only the fast end of the spectrum.

Gemma 4 31B’s global layers (rope_type: "proportional", partial_rotary_factor: 0.25, base 1,000,000, head dimension 512, twice its local layers’ 256) rotate 64 of 256 pairs, whose longest wavelength is 188 tokens; a standard partial RoPE with the same 128 rotated dimensions would reach about 5.1 million (computed). The rotated part of the score in those layers varies only on scales of a few hundred tokens: averaged over thousands it is flat, with no trend from near to far, and the other 384 dimensions contribute the same at every distance. The local layers keep full RoPE at base 10,000 over 1,024-token windows.

Figure 3.The wavelengths each scheme keeps. Rotating a quarter of Gemma 4's global head the standard way would keep wavelengths up to five million tokens; p-RoPE keeps the fastest quarter of the full spectrum, all shorter than 200 tokens, and leaves 192 pairs unrotated.

Barbero et al. trained Gemma 2B models at 8,192 tokens and report perplexity only: p=0.75p = 0.75 beat full RoPE slightly (4.4414 against 4.4627 on Wiki), and p=0.25p = 0.25, Gemma 4’s value, was worse (4.5302; single runs). Long contexts were not tested. Gemma 4’s report states the choice (“p-RoPE with p=0.25 on global attention layers”) and one consequence, a global KV cache “effectively” 37.5% smaller. On our reading that follows from p-RoPE: the global layers reuse the key projection as the value before rotation (attention_k_eq_v: true), so key and value can share storage only where the key isn’t rotated, and 512 + 128 of 1,024 elements is 37.5% less (computed). The report gives no quality reason. Liquid’s LFM3 blog is the one lab document that gives the content-channel reason: global layers “leave the low-frequency half unrotated, so those dimensions match on content regardless of distance” (though LFM3’s released code rotates every dimension).

No position encoding

What the causal mask gives

Remove RoPE from a decoder and it still knows order. Kazemnejad et al. (2023) prove that the first layer can recover absolute positions from the causal mask alone (their Theorem 1). A head whose keys are all equal gives every visible token the same weight 1/t1/t; if its value is 1 for the beginning-of-sequence token, which is always first, and 0 for every other token, it outputs 1/t1/t at position tt, and the feed-forward layer can turn that into a position code. Later layers can then compare positions (Theorem 2). On synthetic tasks with about 107M-parameter models, NoPE generalized to longer inputs better than RoPE, ALiBi (a fixed per-head score penalty linear in distance) or absolute embeddings.

In the paper’s 1B-parameter test, trained at 1,024 tokens, “NoPE and ALiBi generalize similarly to larger context sizes up to almost twice their training context size”, with ALiBi more stable beyond that, while RoPE’s perplexity exploded. NoPE has no parameter tied to a length, but its 1/t1/t signal still reaches values below 1/T01/T_0 that training never showed, and its softmax spreads thinner.

Why it is weak on its own

Locality is costly without explicit position. A single NoPE head whose input carries no position can’t put almost all its weight on the previous token: if the same token appears twice in a row, the query scores both copies equally (Barbero et al., Prop. 5.2). A stack can do it, by first recovering position from the mask and then comparing positions (Kazemnejad et al., Theorems 1 and 2), but it spends layers on what RoPE’s fast pairs give a single head directly. The 1/t1/t signal also gets finer as tt grows: neighbours differ by a relative 1/t1/t, 0.02% at position 4,096 (our arithmetic; the paper doesn’t discuss resolution). In Cohere’s 8B comparison, an all-NoPE model trailed RoPE after fine-tuning, validation loss 1.58 against 1.52 (Yang et al., Table 2). And Qwen tested NoPE in the full-attention layers of its Qwen3.8-Next hybrid and kept RoPE: the two “show little difference during pretraining, but the NoPE variant exhibits a substantially higher rate of endless generation after post-training” (no rates given).

Hybrids: position in some layers, none in others

Local RoPE, global NoPE

Yang et al. (2025) interleaved RoPE and NoPE layers in an 8B model and found “a spontaneous ‘division of labor’”: “RoPE layers focus on local information aggregation and NoPE layers specialize in long-range retrieval”. Their final design gives the RoPE layers a 4,096-token window and base 10,000 and makes every fourth layer a full-attention NoPE layer. Trained to 128K and tested at 256K, it held a RULER retrieval score of 74.8 where the RoPE baseline, its base raised to 8 million, fell to 57.1 (both about 96 at 8K).

A sliding-window layer never sees an offset larger than its window, so its RoPE never extrapolates: Kolibri-1’s local layers see at most 512 tokens back, at any context length.

Figure 4.Where position lives in six hybrid stacks. In five, position sits in windowed or recurrent layers, whose reach is bounded, and the global layers carry no position (Kolibri-1, Kimi K3, GLM-5.3-Flash, Nemotron 3) or only short-wavelength rotation (Gemma 4). Mistral Large 4 keeps YaRN-scaled RoPE in full-context layers and leaves every fourth layer without position.

In Kolibri-1, “Positional information is injected only in the SWA blocks using RoPE with θ = 10 000. The full-attention blocks use no positional encoding”, one block in five (layer_types, 10 of 50). In “early experiments, removing RoPE from the full-attention blocks improved long-context performance”, without numbers. The plotted comparison (Figure 7, a 7.8B-total, 0.6B-active proxy) extends a full-attention RoPE model to 64K with YaRN and trains the hybrid at 256K “without any RoPE extension”, but the arms also differ in attention type and pre-training length, so it compares two designs, not NoPE against RoPE.

The shipped model was trained to 256K and evaluated to a million tokens with no rescaling. On Aleph Alpha’s RULER runs of base models (report, Table 18, the report’s own protocol), Kolibri-1 scores 69.8 at 256K, 65.5 at 512K and 63.2 at 1M. Nemotron 3 Nano scores 72.1, 73.1 and 58.5, and Qwen3.5-35B-A3B, served with static YaRN at factor 4 at every length, 80.1, 72.2 and 57.5. Both lead Kolibri up to 512K and trail it at 1M; Aleph Alpha’s card recommends at most 262,144 tokens for complex tasks.

Mistral Large 4 had sliding-window RoPE layers and one full-attention NoPE layer in four; the surgery removed the windows, and “the original NoPE/global layers remained NoPE” (18 of 72 in the config’s no_rope_layers).

Recurrent layers carry position, attention none

A linear-attention or state-space layer (T2) mixes tokens through a state. A plain running sum would be order-blind; decay gates (a learned, input-dependent form of ALiBi’s distance penalty), the delta rule’s order-dependent updates and short convolutions make these layers order- and recency-sensitive. Three families let those layers carry all position and remove it from every softmax layer:

  • Kimi K3 interleaves 69 Kimi Delta Attention (KDA) layers with 24 MLA layers and “applies No Position Encoding (NoPE) to all MLA layers”; the KDA layers “provide position-sensitive and recency-aware sequence mixing”, which “avoids modifying positional-encoding parameters when extending the context length”.
  • GLM-5.3-Flash interleaves 34 KDA layers with 11 sparse MLA layers and sets qk_rope_head_dim: 0; its code passes position_embeddings=None to every layer. Z.ai’s own card doesn’t mention NoPE; the vLLM recipe calls them “NoPE sparse MLA layers”.
  • Nemotron 3 (Ultra; 3.5 Lightning and Super) puts a few attention layers among Mamba-2 layers: “Since Mamba layers provide implicit positional information, Nemotron 3 models do not use RoPE in attention layers”.

A learned relative bias

Thinking Machines’ Inkling has no RoPE at all. Each query produces, per head, a 16-dimensional vector rtr_t (d_rel: 16), and a learned 16×1,02416 \times 1{,}024 matrix maps it to one score bias per distance: a rank-16, query-dependent relative term in the line of Shaw et al. and Music Transformer (both 2018). The released code limits the bias to the last 1,024 tokens in the 11 global layers (rel_extent: 1024) and 512 in the 55 local ones, so further back a global layer has no explicit position signal, like a NoPE layer. Above 128,000 tokens the global layers also multiply their queries by 1+0.1ln⁡((t+1)/128,000)1 + 0.1\ln((t+1)/128{,}000), 1.21 at 1M (computed), sharpening the softmax as YaRN’s temperature does. The lab says this relative attention “performs better and extrapolates better to longer sequences” than RoPE, without numbers. No mapped model uses ALiBi.

Training at length

Twenty-three of the 35 configs declare 1,048,576 positions (computed), and every documented recipe reaches its length by training there. Kolibri-1 pre-trained 20T tokens at 16K, mid-trained (a stage between pre-training and fine-tuning) 3.4T at 64K and added 200B at 256K; DeepSeek-V4 went 4K, 16K, 64K, 1M; MiMo-V2-Flash changed its base on the way to 256K; Kimi K3 trained at 1M with no position parameter to change.

Without RoPE, a model still has to be trained at the lengths it will serve. In Nemotron 3.5’s report, after mid-training Super at 256K, “the RULER-1M score dropped from 79.9 to 33.5”, and a recovery phase of 9 billion tokens “with the same recipe” as its 1M-context stage brought it to 81.0. How labs schedule these stages, and what data they use, belongs to the training series (S2, and the page on long-context training).

Comparison

MethodWhat changesLayersLocal orderBeyond trained lengthMapped models
RoPErotate pairs by tθit\theta_iall or localstrongslow pairs out of rangenearly all
Larger base (ABF)all but pair 0 slower; trainedglobal or allslightly coarserout of range laterbases 1.5e5 to 1.1e7
PIall θi/κ\theta_i / \kappaallblurredin range to κT0\kappa T_0none
NTK-awarebase ⋅ κdhR/(dhR−2)\cdot\,\kappa^{d_h^R/(d_h^R-2)}allnearly keptslow-middle pairs out by up to ~4×none
YaRN, Llama-3slow pairs /κ/\kappa, fast kept, blend; YaRN adds temperatureglobal or allkeptin range to κT0\kappa T_011 YaRN, 2 Llama-3 type
LeRoPEYaRN times learned factorsRoPE layerskeptlearnedMistral Large 4
Partial RoPErotate dhR<dhd_h^R < d_h dimsall or globalfewer frequenciesrotated dims still need scaling12, plus 12 MLA
p-RoPEkeep fastest fraction ppglobalkeptshort wavelengths onlyGemma 4
NoPEnone; causal mask onlyglobalweak aloneno frequency out of range; still trained at length7
Learned relative biasper-distance score biasallstrong within extentcontent only beyondInkling

Where the models in the map sit

The site checks each model’s released config and modeling code for a list of features (35 language models so far). The table is read from those files, and the counts are cleaned against the code: the stored partial-rope verdict is wrong for Kimi K3 and LFM3, and five unclear YaRN verdicts are absent.

ModelRoPE baseRotated / head dimsScalingNoPE softmax layers
Kolibri-11e4 (SWA)128/128 (SWA)none10 of 50 (all global)
DeepSeek-V3.21e464/192 (MLA)YaRN 40 from 4,096none
DeepSeek-V4, V4.11.6e5 compressed layers; 1e4 sliding-only (Flash layers 0, 1)64/512YaRN 16 from 65,536, compressed layersnone
Gemma 4 31B1e4 SWA, 1e6 global256/256 SWA; global p-RoPE 64 of 256 pairsnonenone
LFM3-1.7B1e4 SWA, 1e6 global128/128 (config says 0.5 global; code rotates all)nonenone
MiniMax-M3, M3.1-Flash5e6, 1e764/128nonenone
Mistral Large 41e4, learned factors64/128 (MLA)YaRN 2,048 from 512 (former window, our reading)18 of 72
Kimi K3none0 (64-dim key slice unrotated)noneall 24 MLA (69 KDA)
GLM-5.3-Flashnone0noneall 11 (34 KDA)
GLM-5.2, 5.38e664/256 (MLA)nonenone
Nemotron 3 Ultra, 3.5 Lightning, Supernone0noneall attention layers (Mamba-2 rest)
gpt-oss-120b1.5e564/64YaRN 32 from 4,096none
Laguna-S-2.11e4 SWA, 5e5 global128/128 SWA, 64/128 globalYaRN 128 from 8,192, globalnone
Qwen3.5, 3.6, 3.81e764/256none (YaRN opt-in)none
Step-3.7-Flash1e4 SWA, 5e6 global128/128 SWA, 64/128 globalLlama-3 type 2 from 131,072, globalnone
Step-5-Preview1e4 SWA, 1e7 global192/192 SWA, 64/192 globalLlama-3 type, factor 1 (no change)none
Hy31.1e7128/128nonenone
Hy4-preview1e764/256 (MLA)nonenone
Inklingno RoPE: learned relative bias0nonen/a
MiMo-V2.5, V2.61e4 SWA, 1e7 global64/192 bothnonenone

The code-check counts, and how they were cleaned:

  • partial-rope is present for 25 of 35 models and 10 of 16 labs after cleaning (raw: 27). Twelve are MLA-style shared rotary keys (DeepSeek, Mistral, Tencent, Z.ai), twelve rotate a fraction of every head (Qwen, Xiaomi, MiniMax, Poolside, StepFun), and one is p-RoPE (Google). Without the MLA-style ones, 13 models from 6 labs chose partial rotation for its own sake.
  • yarn is present for 11 models from 4 labs: DeepSeek (8), OpenAI, Poolside, Mistral; the nine with readable rotary code apply it statically.
  • nope-layers is present for 2 models (Kolibri-1, Mistral Large 4): the check asks for some softmax layers with RoPE and some without, which excludes Kimi K3, GLM-5.3-Flash and the three Nemotron models. Counted together, 7 models from 5 labs have softmax layers without position.
  • Fourteen models set RoPE per layer type. gpt-oss is not one: its sliding and full layers share one RoPE, YaRN included.

Easy to get wrong

  • “Partial RoPE 0.25” means two different spectra. In Qwen3.5 the factor sets dhR=64d_h^R = 64 and the 64 rotated dimensions span wavelengths up to 38 million tokens. In Gemma 4’s global layers the same partial_rotary_factor: 0.25 with rope_type: "proportional" keeps only wavelengths under 188 tokens. Read the rope_type, not the factor.
  • A config value is not a code path: LFM3’s config sets partial_rotary_factor: 0.5 for its global layers, and its released code rotates every dimension.
  • Turning on a model’s optional YaRN changes every key it caches. A prefix cache, or an offloaded KV cache, built with one rope_parameters can’t be reused with another.
  • YaRN’s factor goes on the cosine and sine, scaling only rotated dimensions (all in gpt-oss, half in Laguna’s global layers), on the softmax scale (DeepSeek-V3.2’s mscale_all_dim), or nowhere (DeepSeek-V4, Mistral Large 4).
  • Which coordinates form a pair is a convention: neighbours (2i,2i+1)(2i, 2i+1) in DeepSeek, GLM-5.2 and 5.3 and Mistral Large 4, halves (i,i+dh/2)(i, i + d_h/2) elsewhere. Converting a checkpoint between them means permuting projection weights, and a mismatch fails silently.

Check yourself

  1. gpt-oss has base 150,000, 64 rotary dimensions and T0=4,096T_0 = 4{,}096. Compute the wavelength of pair 12 and the number of turns it completes over T0T_0. What does YaRN do to it?
    Answerθ12=150000−24/64≈0.0115\theta_{12} = 150000^{-24/64} \approx 0.0115 rad per token, so λ12=2π/θ12≈549\lambda_{12} = 2\pi/\theta_{12} \approx 549 tokens and r12=4096/549≈7.5r_{12} = 4096/549 \approx 7.5 turns. That lies between α=1\alpha = 1 and β=32\beta = 32, so pair 12 is in the ramp. With the paper’s ramp, γ=(7.5−1)/31≈0.21\gamma = (7.5 - 1)/31 \approx 0.21 and θ12′/θ12=0.79/32+0.21≈0.23\theta'_{12}/\theta_{12} = 0.79/32 + 0.21 \approx 0.23; Hugging Face’s ramp, linear in the pair index, keeps 0.59.
  2. Kolibri-1’s local layers use RoPE at base 10,000 and are trained at 256K. Why don’t they need rescaling at 1M?
    AnswerThey attend only to the 512 preceding tokens, so the offset Δ\Delta in their rotation never exceeds 512, whatever the context length. Every angle they meet at 1M they met in training.
  3. PI and NTK-aware scaling both bring the slowest pair back into range for κ=32\kappa = 32. Why does PI need fine-tuning while NTK-aware scaling often works without?
    AnswerPI slows the fastest pairs 32-fold too, so adjacent tokens differ by 0.03 rad in pair 0 instead of a radian, a local order the model never saw. NTK-aware scaling keeps pair 0 and leaves pairs 18 to 30 partly out of range instead, which is why fine-tuning with it did worse.
  4. Nemotron 3.5 Super has no RoPE in any layer. Why did its RULER score at 1M still fall after a stage at 256K?
    AnswerNo frequency went out of range, but behaviour at 1M is learned from data at that length; after a stage at 256K it fell (79.9 to 33.5), and 9 billion tokens of long training restored it (81.0).

In the map

Methods

Taxonomy nodes

Further reading

  • Su et al., RoFormer (2021), arXiv:2104.09864: the rotation’s complex-number derivation and the decay bound.
  • Chen et al., Position Interpolation (2023), arXiv:2306.15595: the interpolation bound and recipe.
  • Xiong et al., Effective Long-Context Scaling of Foundation Models (2023), arXiv:2309.16039: the base-frequency change and the 80B-token ablation against PI.
  • Peng et al., YaRN (2023), arXiv:2309.00071: NTK-aware, NTK-by-parts, the temperature fit, dynamic scaling.
  • Kazemnejad et al., The Impact of Positional Encoding on Length Generalization (2023), arXiv:2305.19466: how a causal decoder recovers position.
  • Barbero et al., Round and Round We Go! (2024), arXiv:2410.06205: which frequencies Gemma uses for what, and p-RoPE.
  • Yang et al., Rope to Nope and Back Again (2025), arXiv:2501.18795: the division of labour Kolibri’s design follows.
  • Kolibri-1 technical report §2.1.3, §2.4.3.1, and Kimi K3 technical report §2.1.2, §3.4: the two most detailed lab accounts of a NoPE hybrid.
↑