Xiaomi
MiMo-V2.5
What MiMo-V2.5’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 310B / 15B active
- Context
- 1M
- Attention
- sliding/global hybridhybrid sliding-window (128) + global GQA, learnable attention sink
- Experts
- 256 routed, top-8, no shared
- Input
- text, image, audio, video
- Licence
- MIT
- Released
- 2026-04-22
Against the consensus recipe
Its documents state 8 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionused
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningused
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationcore
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionused
- FP8used
Checked in its code
Its code shows 8 of the 19 architecture features checked — 3 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from the model repository’s own code (modeling_mimo_v2.py, configuration_mimo_v2.py) and config.json at revision 63651580ca, by anthropic/claude-sonnet-5 on 2026-09-25.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | in its code | adopted | Full-attention layers use 64 query heads with 4 KV heads and SWA layers use 64 query heads with 8 KV heads, both grouped via repeat_kv. show evidencenum_attention_heads = 64num_key_value_heads = 4swa_num_key_value_heads = 8 |
| Multi-head latent attention | not in its code | not stated | There is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v… show the whole verdictThere is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v projections. |
| Sliding-window attention | in its code | adopted | sliding_window is 128 and the hybrid_layer_pattern marks most layers as SWA (value 1), enabling create_sliding_window_causal_mask for… show evidencesliding_window is 128 and the hybrid_layer_pattern marks most layers as SWA (value 1), enabling create_sliding_window_causal_mask for those layers. sliding_window = 128hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1… |
| Interleaved sliding-window and global attention | in its code | adopted | hybrid_layer_pattern interleaves full-attention layers (0) every 6 layers with sliding-window layers (1) for the rest, and the decoder… show evidencehybrid_layer_pattern interleaves full-attention layers (0) every 6 layers with sliding-window layers (1) for the rest, and the decoder layer sets attention_type accordingly. hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer scoring mechanism or index_topk/index_n_heads config exists in this codebase. |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All decoder layers use MiMoV2Attention (softmax attention, full or sliding-window); there are no linear-attention or SSM mixer layers. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta rule / recurrent state update mechanism is implemented in the code. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 SSD selective state-space blocks are implemented; no mamba_num_heads or ssm_state_size config exists. |
| Learnable attention sink | in its code | adopted | SWA layers get a learned per-head attention_sink_bias parameter concatenated as an extra logit column in the softmax denominator, enabled… show evidenceSWA layers get a learned per-head attention_sink_bias parameter concatenated as an extra logit column in the softmax denominator, enabled by add_swa_attention_sink_bias=true. add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false |
| Gated attention output | not in its code | not stated | No sigmoid gating of attention output before o_proj is implemented; attn_output goes straight from attention interface to o_proj. |
| QK normalization | not in its code | not stated | No RMSNorm or LayerNorm is applied to query/key states before the dot product in MiMoV2Attention; only rotary embedding is applied. |
| Partial RoPE code only | in its code | not stated | partial_rotary_factor=0.334 makes rope_dim less than head_dim, and the attention forward splits query/key into rope and nope parts… show evidencepartial_rotary_factor=0.334 makes rope_dim less than head_dim, and the attention forward splits query/key into rope and nope parts, applying RoPE only to the rope slice. partial_rotary_factor = 0.334head_dim = 192 |
| YaRN RoPE scaling | not in its code | not stated | rope_scaling type is 'default', not YaRN, so no YaRN scaling is applied. |
| Layers without positional encoding (NoPE) | not in its code | not stated | All softmax attention layers apply RoPE to the rope_dim portion of every head (partial RoPE uniformly applied); there is no per-layer… show the whole verdictAll softmax attention layers apply RoPE to the rope_dim portion of every head (partial RoPE uniformly applied); there is no per-layer toggle disabling RoPE entirely on some attention layers. |
| Mixture of experts | in its code | adopted | MoE layers route each token via a sigmoid-scored router (noaux_tc) to 8 of 256 routed experts per token, per moe_layer_freq. show evidencen_routed_experts = 256num_experts_per_tok = 8moe_layer_freq = [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1… |
| Shared expert | not in its code | not stated | n_shared_experts is null and MiMoV2MoE has no shared-expert module in the code. show evidencen_shared_experts = null |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The gate adds e_score_correction_bias to scores only for top-k selection (noaux_tc), while the actual weights used to combine expert… show evidenceThe gate adds e_score_correction_bias to scores only for top-k selection (noaux_tc), while the actual weights used to combine expert outputs come from the unbiased sigmoid scores. topk_method = "noaux_tc" |
| Multi-token prediction layers | code unclear | adopted | There is a regex to ignore loading model.mtp.* weights, implying an MTP module may exist, but no num_nextn_predict_layers or similar… show evidenceThere is a regex to ignore loading model.mtp.* weights, implying an MTP module may exist, but no num_nextn_predict_layers or similar config key or module implementation is present in the given files. |
| SwiGLU feed-forward code only | in its code | not stated | MLP and expert FFNs compute down_proj(silu(gate_proj(x)) * up_proj(x)) with hidden_act set to silu. show evidencehidden_act = "silu" |
token mixer 8
softmax attention 7
sliding window attention 3
“Sliding Window Attention (SWA)”
“through its hybrid Sliding Window Attention architecture, lightweight Multi-Token Prediction, and the MOPD post-training paradigm.”
“SWA-128”
grouped-query attention 1
“Both SWA and GA utilize Grouped-Query Attention (GQA)”
attention sink 1
“learnable attention sink bias”
hybrid layer stacking 1
“MiMo-V2-Flash adopts a hybrid attention architecture that interleaves Sliding Window Attention (SWA) with global attention”
channel mixer 6
dense feed-forward network 1
“MTP block uses dense FFN instead of MoE”
mixture of experts 5
“MoE models suffer from inconsistent expert routing across rollout and training due to numerical precision issues”
expert routing 1
“it uses 256 routed experts (top-8)”
expert load balancing 3
“we reference historical pass rates and, if necessary, assign new prompts to GPUs with load balancing”
“The expert bias update factor is set to 0.001”
“--load-balance-method round_robin”
positional encoding 1
“Rotary Positional Embedding (RoPE) is partially applied to the first 64 dimensions query and key”
prediction head 2
“We analyze the relationship between the model’s predictive uncertainty measured by next token cross-entropy and the efficiency of the…”
“by repurposing MTP as a draft model for speculative decoding”
multimodal architecture 6
“MiMo-V2.5 is natively omnimodal — trained from scratch to see, hear, and act across modalities”
“With native visual and audio understanding, MiMo-V2.5 reasons seamlessly across modalities”
“Equipped with a 729M-param Vision Transformer (ViT) featuring hybrid window attention”
“augmented with dedicated visual and audio encoders (both pretrained in-house) connected through lightweight projectors”
“dedicated audio encoder initialized from the weights of MiMo-Audio”
“projector warmup to align the audio and visual projectors with the language model”
context capacity 2
“Its 1M context window supports complete documents, extended conversations, and complex task contexts in a single pass”
“during which the context window is progressively extended from 32K to 256K to 1M”
Not used 1
“and contains no shared experts.”
language modelling objective 1
“Training goes through five stages: text pre-training on diverse corpora”
multi-token prediction objective 1
“We support MTP in vLLM and enhance the robustness of the inference engine in the RL system.”
distillation objective 1
“reverse KL divergence loss between student and teacher”
auxiliary loss 1
“the MoE sequence auxiliary loss coefficient is set to 0.01”
learning-rate schedule 1
“cosine decay to”
training precision 1
“we adopt an FP8 mixed-precision framework similar to DeepSeek-V3”
training stability 1
“Gradient clipping is applied with a maximum norm of 1.0”
data filtering 1
“applying multi-dimensional data filtering to increase reasoning pattern density in pre-training data”
data mixture & curriculum 1
“We adopt a three-stage data mixture strategy for pre-training.”
supervised fine-tuning 2
“Post-training incorporates SFT, large-scale agentic RL, and Multi-Teacher On-Policy Distillation (MOPD)”
“we perform RL training on a cold-started SFT model, resulting in MiMo-7B-RL”
reinforcement learning algorithm 3
“large-scale Reinforcement Learning (RL)”
“We propose Rollout Routing Replay (R3) [ma2025stabilizing] to train RL using the same routed experts from rollout”
reward modelling 3
“Outcome Reward Models (ORMs)”
“Each problem undergoes careful cleaning and difficulty assessment to ensure quality. We employ only rule-based accuracy rewards to avoid…”
“To mitigate the sparse reward issue for challenging code problems, we introduce a test difficulty driven code reward.”
policy distillation 2
“through its hybrid Sliding Window Attention architecture, lightweight Multi-Token Prediction, and the MOPD post-training paradigm.”
“on-policy distillation using dense, token-level rewards from teacher logits”
rollout & RL infrastructure 8
“We integrate partial rollout [team2025kimi, fu2025areal] to partition overlong trajectories across steps”
“we extend the Seamless Rollout Engine [xia2025mimo] and implement a Data Scheduler”
“Our design integrates continuous rollout, asynchronous reward computation, and early termination to minimize GPU idle time”
“We implement a data re-sampling strategy for easy problems to enhance rollout sampling efficiency and stabilize policy updates…”
“we extend the Seamless Rollout Engine [xia2025mimo] and implement a Data Scheduler to seamlessly schedule fine-grained sequences instead…”
“By employing staleness-aware truncated importance sampling for partial rollout, we significantly accelerate RL training without…”
“Tool Manager coordinates with Toolbox to accelerate training through environment pre-warming and sequence-level asynchronous reward…”
“We implement Toolbox and Tool Manager to tackle global resource contention and local inefficiency in RL agent training.”
agentic post-training 2
“Post-training incorporates SFT, large-scale agentic RL, and Multi-Teacher On-Policy Distillation (MOPD)”
“supervised fine-tuning and agentic post-training”
decoding strategy 3
“We support MTP in vLLM and enhance the robustness of the inference engine in the RL system.”
“accelerate inference via speculative decoding”
“--speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1”
KV cache management 2
“--chunked-prefill-size 16384”
“we employ a request-level prefix cache during rollout. This cache stores KVCache and MoE routed experts from prior turns”
inference quantization 1
“ships with native FP8 (block-wise e4m3) weights”
serving parallelism 2
“--moe-a2a-backend deepep”
“--enable-dp-attention”
inference kernel 1
“--attention-backend fa3 --mm-attention-backend fa3”
context management 2
“replicating their aggressive reset protocol, we achieve 58.3 on comparable benchmarks.”
“we combat the 'Lost in the Middle' phenomenon by enforcing aggressive memory compression. When context utilization exceeds a threshold ...…”
agentic scaffolding 1
“we adopt a Unix-inspired abstraction: tools, documents, and databases are uniformly exposed as files, enabling the model to retrieve…”
Not used 1
KV cache management “Unlike the commonly-used radix cache in current inference engines, our request-level prefix cache avoids re-prefilling or inter-request…”
agent product 3
“Xiaomi explicitly positions it as a drop-in backend for scaffolds including Claude Code, OpenCode, and Kilo”
“Xiaomi explicitly positions it as a drop-in backend for scaffolds including Claude Code, OpenCode, and Kilo”
“Xiaomi explicitly positions it as a drop-in backend for scaffolds including Claude Code, OpenCode, and Kilo”
unfiled 4
“MiMo-V2.5-Pro is Xiaomi’s most capable model to date, delivering significant improvements over its predecessor, MiMo-V2-Pro, in general…”
“it makes full use of the affordances of its harness environment, manages its memory, and shapes how its own context is populated toward…”
“multimodal pre-training at scale on high-quality cross-modal data”
“Wired into an ngspice simulation loop, in about an hour of closed-loop iteration — calling the simulator, reading waveforms, tweaking…”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- MiMo-V2.5 architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| mimo.xiaomi.com/mimo-v2-5 | official blog | Xiaomi MiMo | read |
| huggingface.co/XiaomiMiMo/MiMo-V2.5 | model card | Xiaomi MiMo | read |
| huggingface.co/XiaomiMiMo/MiMo-V2.5-Base | model card | Xiaomi MiMo | read |
| github.com/XiaomiMiMo/MiMo | code repo | Xiaomi MiMo | read |
| arxiv.org/abs/2601.02780 | technical report | Xiaomi LLM-Core | read |
| recipes.vllm.ai/XiaomiMiMo/MiMo-V2.5 | vendor docs | vLLM | read |
| marktechpost.com/2026/04/22/xiaomi-releases-mimo-v2-5-pro-and-mimo-v2-5… | news | MarkTechPost | read |
| openrouter.ai/xiaomi/mimo-v2.5 | vendor docs | OpenRouter | read |