gpt-oss-120b
OpenAIextend the context length of dense layers to tokens using YaRN
specific method · filed under model architecture
YaRN is a positional-encoding method that extends a model's context length by rescaling the RoPE frequencies.1
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
used YaRN to rescale the RoPE frequencies— technical report
Also called YaRN context length extension, YaRN context window extension, YaRN RoPE scaling, YaRN RoPE scaling for long context, YaRN rotary-position-frequency rescaling, YaRN (Yet another RoPE extensioN).
Checked as “YaRN RoPE scaling” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| Mistral-Large-4 | in its code | discussed | rope_parameters={"attention_factor": 1.0, "beta_fas…rope_parameters.rope_type is 'yarn' and the rotary embedding module dispatches to the YaRN init function via ROPE_INIT_FUNCTIONS, with mscale applied to attention scaling. text_config.rope_parameters = {"attention_factor": 1.0, "beta_fast": 32, "beta_slow": 1, "factor": 2048, "o… | code → |
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | rope_scaling={"rope_type": "yarn", "factor": 16,…rope_scaling in text_config sets rope_type to yarn with factor 16 and related YaRN parameters, applied via the attention module's rotary_emb (cos_sin_cache) referenced throughout the code. text_config.rope_scaling = {"rope_type": "yarn", "factor": 16, "beta_fast": 32, "beta_slow": 1, "origina… | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | rope_scaling={"rope_type": "yarn", "factor": 16,…rope_scaling in text_config sets rope_type to yarn with factor 16 and associated beta parameters, and the model's rotary embedding (rotary_emb.cos_sin_cache) consumes this config. text_config.rope_scaling = {"rope_type": "yarn", "factor": 16, "beta_fast": 32, "beta_slow": 1, "origina… | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | rope_scaling={"beta_fast": 32, "beta_slow": 1, "…rope_scaling type is yarn and is applied to the compress rope-type branch (CSA/HCA layers) via ROPE_INIT_FUNCTIONS dispatch keyed by rope_type. rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | rope_scaling={"beta_fast": 32, "beta_slow": 1, "…rope_scaling type is yarn and is applied to the compress-branch rope used by CSA/HCA attention layers via the ROPE_INIT_FUNCTIONS dispatch keyed by rope_type rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | rope_scaling={"beta_fast": 32, "beta_slow": 1, "…rope_scaling specifies type yarn with factor 16 and this config's compress-branch RoPE parameters set rope_type yarn, which the rotary embedding dispatches to the YaRN init function. rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… | code → |
| Laguna-S-2.1 | in its code | not stated | rope_type=yarn · factor=128.0The full_attention rope_parameters entry sets rope_type yarn with factor/beta_fast/beta_slow, and the rotary embedding dispatches to ROPE_INIT_FUNCTIONS for non-default types. rope_parameters.full_attention.rope_type = "yarn"rope_parameters.full_attention.factor = 128.0 | code → |
| DeepSeek-V4-Flash | in its code | not stated | rope_scaling={"beta_fast": 32, "beta_slow": 1, "…rope_scaling type is yarn in the config and compress-branch rope_parameters are built with this yarn configuration (factor 16, beta_fast/slow) consumed by ROPE_INIT_FUNCTIONS for the 'compress' layer type. rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… | code → |
| DeepSeek-V4-Pro | in its code | not stated | rope_scaling={"beta_fast": 32, "beta_slow": 1, "…rope_scaling.type is yarn and is wired into the compress rope-type parameters that the YaRN-capable ROPE_INIT_FUNCTIONS dispatch consumes for compressed-attention layers. rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… | code → |
| DeepSeek-V3.2 | in its code | not stated | type=yarn · factor=40rope_scaling.type is 'yarn' and the rotary embedding init dispatches to the YaRN ROPE_INIT_FUNCTIONS, with mscale applied to the attention scaling. rope_scaling.type = "yarn"rope_scaling.factor = 40 | code → |
| gpt-oss-120b | in its code | adopted | rope_type=yarn · factor=32.0rope_scaling specifies rope_type yarn with factor 32.0, and the rotary embedding module dispatches to the YaRN init function rope_scaling.rope_type = "yarn"rope_scaling.factor = 32.0 | code → |
| LFM3-1.7B | not in its code | not stated | rope_parameters.full_attention.rope_type=default · rope_parameters.sliding_attention.rope_type=defaultrope_parameters entries specify rope_type "default" for both layer types, not YaRN, and no YaRN scaling code path is exercised. rope_parameters.full_attention.rope_type = "default"rope_parameters.sliding_attention.rope_type = "default" | code → |
| MiniMax-M3.1-Flash | code unclear | not stated | verdict and linesrope_parameters is not populated with an explicit rope_scaling/type in the given config (only rope_theta and rotary_dim appear at top level of text_config), so whether YaRN is switched on cannot be determined from what is given. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | not in its code | not stated | verdict and linesNo rope_scaling or rope_parameters key is present anywhere in the config, so no YaRN scaling is configured. | code → |
| Step-5-Preview | not in its code | not stated | rope_scaling={"rope_type": "llama3", "factor": 1…rope_scaling.rope_type is 'llama3', not YaRN, so no YaRN scaling is applied despite yarn_only_types gating logic in code. text_config.rope_scaling = {"rope_type": "llama3", "factor": 1.0, "original_max_position_embeddings": 10… | code → |
| Kolibri-1 | code unclear | not stated | rope_theta=10000.0rope_parameters is read from config but its contents (rope_type, possible yarn scaling) are not shown in the given config.json, which only lists rope_theta at top level. rope_theta = 10000.0 | code → |
| MiMo-V2.6-Flash | not in its code | not stated | rope_type=default · type=defaultrope_parameters/rope_type is set to "default" in this config, not YaRN, so no YaRN scaling is applied. rope_parameters.rope_type = "default"rope_parameters.type = "default" | code → |
| MiMo-V2.6-Pro | not in its code | not stated | rope_type=default · type=defaultrope_parameters/rope_type is set to "default", not YaRN, so no YaRN scaling is applied. rope_parameters.rope_type = "default"rope_parameters.type = "default" | code → |
| Hy4-preview | not in its code | not stated | rope_parameters={"rope_theta": 10000000, "rope_type…rope_parameters.rope_type is set to 'default' rather than 'yarn', so although yarn-related mscale helper code exists it is not activated by this config. rope_parameters = {"rope_theta": 10000000, "rope_type": "default"} | code → |
| GLM-5.3-Flash | not in its code | not stated | verdict and linesNo rope_scaling or rope_parameters key of type YaRN appears in text_config, and the text attention path uses NoPE (no rotary applied at all), so YaRN scaling cannot apply. | code → |
| Qwen3.8-Flash-Next | not in its code | adopted | rope_type=defaultrope_parameters.rope_type is set to "default", not YaRN, so no YaRN scaling is applied. text_config.rope_parameters.rope_type = "default" | code → |
| GLM-5.3 | not in its code | not stated | rope_parameters={"rope_theta": 8000000, "rope_type"…rope_parameters.rope_type is "default", not yarn, so although yarn_apply_mscale code exists it is a no-op for this config. rope_parameters = {"rope_theta": 8000000, "rope_type": "default"} | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | verdict and linesNo rope_scaling key is present in this config, so no YaRN scaling is configured. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | rope_scaling={"mrope_interleaved": true, "mrope_…rope_scaling in text_config sets rope_type to 'default' with mrope_section/mrope_interleaved for multimodal RoPE, not YaRN, so YaRN scaling is not enabled. vlm_config.text_config.rope_scaling = {"mrope_interleaved": true, "mrope_section": [24, 20, 20], "rope_type": "defa… | code → |
| Kimi K3 | code unclear | not stated | verdict and linesThe config exposes a rope_scaling attribute in KimiLinearConfig defaulting to None and it is not present as a key in the given text_config, and no rotary_emb / RoPE application code is shown (self.rotary_emb is set to None and never used), so YaRN scaling cannot be confirmed as active. | code → |
| Inkling | not in its code | not stated | verdict and linesThere is no rope_scaling key in the config and no RoPE scaling code of any kind; positions are encoded via a learned relative-bias projection, not rotary embeddings. | code → |
| Hy3 | not in its code | not stated | rope_parameters={"rope_theta": 11158840.0, "rope_ty…rope_parameters.rope_type is set to 'default', not yarn, so no YaRN scaling is applied. rope_parameters = {"rope_theta": 11158840.0, "rope_type": "default"} | code → |
| GLM-5.2 | not in its code | not stated | rope_parameters={"rope_theta": 8000000, "rope_type"…rope_parameters.rope_type is "default" (not yarn), so although yarn_apply_mscale code exists, it is a no-op for this config. rope_parameters = {"rope_theta": 8000000, "rope_type": "default"} | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | code unclear | not stated | verdict and linesNo rope_scaling or rope_parameters key is present in the config, and the attention module doesn't reference rotary embeddings or scaling at all, so YaRN cannot be confirmed or ruled out from the given files. | code → |
| MiniMax-M3 | code unclear | not stated | verdict and linesrope_parameters is not given a literal value in the provided config.json (it is None in the class default) so whether YaRN scaling is active cannot be determined from these files. | code → |
| Step-3.7-Flash | not in its code | not stated | rope_scaling={"rope_type": "llama3", "factor": 2…rope_scaling specifies rope_type 'llama3' (not YaRN) and the config has no YaRN rope_type entry; the 'yarn_only_types' key only names which layer types the llama3-style scaling applies to. text_config.rope_scaling = {"rope_type": "llama3", "factor": 2.0, "original_max_position_embeddings": 13… | code → |
| MolmoAct 2 | not in its code | not stated | rope_parameters={"rope_theta": 5000000.0, "rope_typ…rope_parameters.rope_type is 'default' (no YaRN scaling type set), so the RoPE scaling code path for YaRN is not switched on. text_config.rope_parameters = {"rope_theta": 5000000.0, "rope_type": "default"} | code → |
| MiMo-V2.5 | not in its code | not stated | rope_scaling={"rope_type": "default", "type": "d…rope_scaling type is 'default', not YaRN, so no YaRN scaling is applied. rope_scaling = {"rope_type": "default", "type": "default"} | code → |
| MiMo-V2.5-Pro | not in its code | not stated | verdict and linesrope_scaling is not set in this config (no key present) so no YaRN scaling is applied; rotary embedding falls back to the default type. | code → |
| Qwen3.6-35B-A3B | not in its code | adopted | rope_type=defaultrope_parameters.rope_type is "default", not yarn, so no YaRN scaling is applied. text_config.rope_parameters.rope_type = "default" | code → |
| Gemma 4 31B | not in its code | not stated | rope_parameters={"full_attention": {"partial_rotary…rope_parameters only declare rope_type 'proportional' and 'default', not YaRN, so no YaRN scaling is configured or applied. text_config.rope_parameters = {"full_attention": {"partial_rotary_factor": 0.25, "rope_theta": 1000000.0, "… | code → |
| Qwen3.5-397B-A17B | not in its code | adopted | rope_parameters={"mrope_interleaved": true, "mrope_…rope_parameters.rope_type is "default", not "yarn", so no YaRN scaling is switched on. text_config.rope_parameters = {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "defa… | code → |
| X-VLA | not in its code | not stated | verdict and linesThere is no rope_scaling key in the config and no RoPE-scaling code (YaRN or otherwise) anywhere in the provided source, consistent with the model using learned absolute positions instead of RoPE. | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | model_type=llamaThere is no rope_scaling or rope_parameters key in text_config, so no YaRN scaling is configured. text_config.model_type = "llama" | code → |
11 quotes from 11 sources, one card per model, each opening with its strongest.
extend the context length of dense layers to tokens using YaRN
For longer inputs, apply YaRN RoPE scaling via --hf-overrides and raise --max-model-len.
apply YaRN RoPE scaling via --hf-overrides and raise --max-model-len
we recommend using RoPE scaling techniques to handle long texts effectively., e.g., YaRN.
we recommend using RoPE scaling techniques to handle long texts effectively., e.g., YaRN.
extensible to 1M via YaRN
For long-horizon tasks where the total length (including both input and output) exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively, e.g., YaRN.
YaRN is currently supported by several inference frameworks, e.g., transformers, vllm, ktransformers and sglang.
For longer inputs, apply YaRN RoPE scaling via --hf-overrides and raise --max-model-len.
For long-horizon tasks where the total length (including both input and output) exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively, e.g., YaRN.
we removed the attention windows and used YaRN to rescale the RoPE frequencies by 8×
Picked by hand, not extracted: where to read more, not evidence for anything on this page.