Model techniques map
ModelsMiMo-V2.5

Xiaomi · usage rank #8 · released 2026-04-22

MiMo-V2.5

What MiMo-V2.5’s own documents say it is built from — every method with the sentence that describes it.

OpenRouterHugging Face

Curator’s note310B total / 15B active omnimodal MoE, 1M context, MIT license. No dedicated arXiv report found for V2.5; the MiMo-V2-Flash report is the closest architecture lineage document.

Rank by open-weight tokens on OpenRouter, week of 2026-09-21.

Against the consensus recipe

Its documents state 9 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 8 of the 19 architecture features checked — 3 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from the model repository’s own code (modeling_mimo_v2.py, configuration_mimo_v2.py) and config.json at revision 63651580ca, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsEvidence
Grouped-query attention in its code used

Full-attention layers use 64 query heads with 4 KV heads and SWA layers use 64 query heads with 8 KV heads, both grouped via repeat_kv.

num_attention_heads = 64num_key_value_heads = 4swa_num_key_value_heads = 8
self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
modeling_mimo_v2.py · L259
Multi-head latent attention not in its codenot stated

There is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v projections.

Sliding-window attention in its code core

sliding_window is 128 and the hybrid_layer_pattern marks most layers as SWA (value 1), enabling create_sliding_window_causal_mask for those layers.

sliding_window = 128hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1…
self.sliding_window = getattr(config, "sliding_window", None) if is_swa else None
modeling_mimo_v2.py · L262
Interleaved sliding-window and global attention in its code core

hybrid_layer_pattern interleaves full-attention layers (0) every 6 layers with sliding-window layers (1) for the rest, and the decoder layer sets attention_type accordingly.

hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1…
is_swa_layer = config.hybrid_layer_pattern[layer_idx] == 1
        self.attention_type = "sliding_window_attention" if is_swa_layer else "full_attention"
modeling_mimo_v2.py · L400–401
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer scoring mechanism or index_topk/index_n_heads config exists in this codebase.

Linear-attention or state-space layers alongside full attention not in its codenot stated

All decoder layers use MiMoV2Attention (softmax attention, full or sliding-window); there are no linear-attention or SSM mixer layers.

Gated DeltaNet layers not in its codenot stated

No gated delta rule / recurrent state update mechanism is implemented in the code.

Mamba-2 layers not in its codenot stated

No Mamba-2 SSD selective state-space blocks are implemented; no mamba_num_heads or ssm_state_size config exists.

Learnable attention sink in its code used

SWA layers get a learned per-head attention_sink_bias parameter concatenated as an extra logit column in the softmax denominator, enabled by add_swa_attention_sink_bias=true.

add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false
if sinks is not None:
        sinks = module.attention_sink_bias.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
        attn_weights = torch.cat([attn_weights, sinks], dim=-1)
modeling_mimo_v2.py · L89–91
self.attention_sink_bias = (
            nn.Parameter(torch.empty(self.num_attention_heads), requires_grad=False)
            if (
                (getattr(config, "add_full_attention_sink_bias", False) and not is_swa)
                or (getattr(config, "add_swa_attention_sink_bias", False) and is_swa)
            )
            else None
        )
modeling_mimo_v2.py · L268–275
Gated attention output not in its codenot stated

No sigmoid gating of attention output before o_proj is implemented; attn_output goes straight from attention interface to o_proj.

QK normalization not in its codenot stated

No RMSNorm or LayerNorm is applied to query/key states before the dot product in MiMoV2Attention; only rotary embedding is applied.

Partial RoPE code only in its codenot stated

partial_rotary_factor=0.334 makes rope_dim less than head_dim, and the attention forward splits query/key into rope and nope parts, applying RoPE only to the rope slice.

partial_rotary_factor = 0.334head_dim = 192
query_rope, query_nope = query_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
        key_rope, key_nope = key_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
modeling_mimo_v2.py · L306–307
YaRN RoPE scaling not in its codenot stated

rope_scaling type is 'default', not YaRN, so no YaRN scaling is applied.

rope_scaling = {"rope_type": "default", "type": "default"}
Layers without positional encoding (NoPE) not in its codenot stated

All softmax attention layers apply RoPE to the rope_dim portion of every head (partial RoPE uniformly applied); there is no per-layer toggle disabling RoPE entirely on some attention layers.

Mixture of experts in its code core

MoE layers route each token via a sigmoid-scored router (noaux_tc) to 8 of 256 routed experts per token, per moe_layer_freq.

n_routed_experts = 256num_experts_per_tok = 8moe_layer_freq = [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
self.experts = nn.ModuleList(
            [MiMoV2MLP(config, intermediate_size=config.moe_intermediate_size) for _ in range(config.n_routed_experts)]
        )
modeling_mimo_v2.py · L191–193
Shared expert not in its codenot stated

n_shared_experts is null and MiMoV2MoE has no shared-expert module in the code.

n_shared_experts = null
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The gate adds e_score_correction_bias to scores only for top-k selection (noaux_tc), while the actual weights used to combine expert outputs come from the unbiased sigmoid scores.

topk_method = "noaux_tc"
scores_for_choice = scores.view(bsz * seq_len, -1) + self.e_score_correction_bias.unsqueeze(0)
modeling_mimo_v2.py · L164
_, topk_idx = torch.topk(tmp_scores, k=self.top_k, dim=-1, sorted=False)
            topk_weight = scores.gather(1, topk_idx)
modeling_mimo_v2.py · L175–176
Multi-token prediction layers code unclear core

There is a regex to ignore loading model.mtp.* weights, implying an MTP module may exist, but no num_nextn_predict_layers or similar config key or module implementation is present in the given files.

r"model\.mtp\..*",
modeling_mimo_v2.py · L1694
SwiGLU feed-forward code only in its codenot stated

MLP and expert FFNs compute down_proj(silu(gate_proj(x)) * up_proj(x)) with hidden_act set to silu.

hidden_act = "silu"
return self.down_proj(self.act_fn(self.gate_proj(hidden_states)) * self.up_proj(hidden_states))
modeling_mimo_v2.py · L132

model architecture 26

token mixer 8

softmax attention 7

global attention 2

Global attention core
Full attention used

grouped-query attention 1

Grouped-query attention used

attention sink 1

Learnable attention sink bias used

hybrid layer stacking 1

Hybrid Attention core

channel mixer 8

dense feed-forward network 1

Dense feed-forward network used

mixture of experts 7

Mixture of Experts core
MoE with 256 experts and top-8 routing used

expert routing 1

Top-8 expert routing used

shared experts 1

No Shared Experts not used

positional encoding 1

Rotary Position Embedding used

prediction head 1

Multi-Token Prediction core

multimodal architecture 6

Native multimodal understanding core
Native visual and audio understanding core
Vision Transformer (ViT) used
Audio encoder initialized from MiMo-Audio used
Projector warmup used
Visual and audio encoders with lightweight projectors used

context capacity 2

1M-token context window used
Progressive context extension used

training objective 4

language modelling objective 1

Text pre-training used

multi-token prediction objective 1

Multi-Token Prediction used

distillation objective 1

Reverse KL divergence used

auxiliary loss 1

MoE sequence auxiliary loss used

optimization 4

optimizer 1

AdamW used

learning-rate schedule 1

Cosine decay used

training precision 1

FP8 mixed-precision training used

training stability 1

Gradient clipping used

data curation 2

data filtering 1

Multi-dimensional data filtering used

data mixture & curriculum 1

Three-stage data mixture strategy used

post-training 20

supervised fine-tuning 2

Supervised fine-tuning used
Cold-started SFT model used

reinforcement learning algorithm 3

Group Relative Policy Optimization used
Reinforcement Learning used
Rollout Routing Replay used

reward modelling 3

Outcome Reward Model used
Rule-based verifier used
Test-difficulty-driven code reward used

policy distillation 2

Multi-Teacher On-Policy Distillation core
On-Policy Distillation used

rollout & RL infrastructure 8

Partial rollout used
Seamless Rollout Engine used
Continuous rollout with asynchronous reward computation and early termination used
Data re-sampling strategy used
Data Scheduler used
Staleness-aware truncated importance sampling used
Tool Manager used
Toolbox used

agentic post-training 2

Agentic reinforcement learning used
Agentic post-training used

inference & serving 14

decoding strategy 3

Speculative decoding used
Multi-Token Prediction used
EAGLE optional

KV cache management 3

Chunked prefill used
Request-level prefix cache used
RadixCache not used

inference quantization 2

FP8 used
Block-wise E4M3 FP8 weight quantization used

serving parallelism 2

DeepEP used
Attention data parallelism optional

inference kernel 1

FlashAttention 3 used

context management 2

Discarding tool-call history used
Memory compression used

agentic scaffolding 1

Bash commands for context retrieval used

software implementation 3

agent product 3

OpenCode optional
Claude Code optional
Kilo optional

unfiled 4

agentic model core
harness awareness core
multimodal pre-training used
ngspice simulation used

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

DocumentKindPublisherStatus
mimo.xiaomi.com/mimo-v2-5official blogXiaomi MiMoread
huggingface.co/XiaomiMiMo/MiMo-V2.5model cardXiaomi MiMoread
huggingface.co/XiaomiMiMo/MiMo-V2.5-Basemodel cardXiaomi MiMoread
github.com/XiaomiMiMo/MiMocode repoXiaomi MiMoread
arxiv.org/abs/2601.02780technical reportXiaomi LLM-Coreread
recipes.vllm.ai/XiaomiMiMo/MiMo-V2.5vendor docsvLLMread
marktechpost.com/2026/04/22/xiaomi-releases-mimo-v2-5-pro-and-mimo-v2-5…newsMarkTechPostread
openrouter.ai/xiaomi/mimo-v2.5vendor docsOpenRouterread