Xiaomi
MiMo-V2.5-Pro
What MiMo-V2.5-Pro’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 1.02T / 42B active
- Context
- 1M
- Attention
- sliding/global hybridhybrid sliding-window (128) + global GQA, learnable attention sink
- Experts
- 384 routed, top-8, no shared
- Input
- text
- Licence
- MIT
- Released
- 2026-04-22
Against the consensus recipe
Its documents state 6 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionused
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionused
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningused
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationcore
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionused
- FP8not in its documents
Checked in its code
Its code shows 8 of the 19 architecture features checked — 4 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from the model repository’s own code (modeling_mimo_v2.py, configuration_mimo_v2.py) and config.json at revision 21d1ecfecd, by anthropic/claude-sonnet-5 on 2026-09-25.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | Full attention layers use 128 query heads sharing 8 key/value heads via repeat_kv grouping. show evidencenum_attention_heads = 128num_key_value_heads = 8 |
| Multi-head latent attention | not in its code | not stated | No latent KV down/up projection mechanism (kv_lora_rank/q_lora_rank) exists in the code or config; attention uses standard per-head q/k/v… show the whole verdictNo latent KV down/up projection mechanism (kv_lora_rank/q_lora_rank) exists in the code or config; attention uses standard per-head q/k/v projections. |
| Sliding-window attention | in its code | adopted | SWA layers (marked by hybrid_layer_pattern==1) use sliding_window=128 to build a sliding-window causal mask. show evidencesliding_window = 128 |
| Interleaved sliding-window and global attention | in its code | adopted | hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across 70 layers. show evidence |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer-scoring or top-k token selection mechanism appears in the code or config (no index_topk/index_n_heads keys). |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All decoder layers use MiMoV2Attention (softmax attention, either full or sliding-window); there is no linear-attention or SSM mixer layer… show evidenceAll decoder layers use MiMoV2Attention (softmax attention, either full or sliding-window); there is no linear-attention or SSM mixer layer type. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta-rule recurrent mixer implementation or config keys (linear_num_key_heads, linear_conv_kernel_dim) are present. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 SSD state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present. |
| Learnable attention sink | in its code | adopted | Sliding-window layers add a learned per-head attention_sink_bias as an extra logit column in the softmax denominator (enabled via… show evidenceSliding-window layers add a learned per-head attention_sink_bias as an extra logit column in the softmax denominator (enabled via add_swa_attention_sink_bias=true). add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false |
| Gated attention output | not in its code | not stated | No sigmoid gating multiplication is applied to the attention output before o_proj; attn_output goes directly into o_proj. show evidence |
| QK normalization | not in its code | not stated | No RMSNorm/LayerNorm is applied to query or key states before the attention dot product in MiMoV2Attention; queries/keys go straight from… show the whole verdictNo RMSNorm/LayerNorm is applied to query or key states before the attention dot product in MiMoV2Attention; queries/keys go straight from projection/RoPE into matmul. |
| Partial RoPE code only | in its code | not stated | Only rope_dim = head_dim * partial_rotary_factor (0.334) of each head is rotated, the remainder (query_nope/key_nope) carries no position… show evidenceOnly rope_dim = head_dim * partial_rotary_factor (0.334) of each head is rotated, the remainder (query_nope/key_nope) carries no position signal. partial_rotary_factor = 0.334 |
| YaRN RoPE scaling | not in its code | not stated | rope_scaling is not set in this config (no key present) so no YaRN scaling is applied; rotary embedding falls back to the default type. show evidence |
| Layers without positional encoding (NoPE) | not in its code | not stated | All softmax-attention layers apply the same partial_rotary_factor RoPE scheme uniformly (full-attention and sliding-window layers both get… show evidenceAll softmax-attention layers apply the same partial_rotary_factor RoPE scheme uniformly (full-attention and sliding-window layers both get rotary embeddings via rotary_emb/swa_rotary_emb); there is no layer subset entirely without positional encoding. |
| Mixture of experts | in its code | adopted | Feed-forward layers route tokens to 8-of-384 experts chosen by a learned sigmoid router (noaux_tc top-k selection). show evidencen_routed_experts = 384num_experts_per_tok = 8 |
| Shared expert | not in its code | not stated | n_shared_experts is null in this config and no shared-expert module is instantiated in MiMoV2MoE. show evidencen_shared_experts = null |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method noaux_tc), not for weighting. show evidencetopk_method = "noaux_tc" |
| Multi-token prediction layers | code unclear | adopted | The code ignores model.mtp.* weights on load and the config has no nextn/mtp layer count key, so it cannot be determined whether MTP… show evidenceThe code ignores model.mtp.* weights on load and the config has no nextn/mtp layer count key, so it cannot be determined whether MTP modules are declared for this checkpoint. |
| SwiGLU feed-forward code only | in its code | not stated | MLP and expert feed-forwards compute silu(gate_proj(x)) * up_proj(x) then down_proj, with hidden_act set to silu. show evidencehidden_act = "silu" |
token mixer 5
softmax attention 4
global attention 1
“Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 6:1 ratio”
sliding window attention 2
“Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 6:1 ratio”
“Its language backbone inherits from MiMo-V2-Flash's hybrid sliding-window attention architecture”
attention sink 1
“via learnable attention sink bias”
hybrid layer stacking 1
“Hybrid Attention Architecture: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 6:1 ratio and 128 sliding window.”
channel mixer 1
mixture of experts 1
“MiMo-V2.5-Pro is a Mixture of Experts (MoE) model with 1.0 trillion total parameters, but only 42 billion active parameters are used…”
prediction head 2
“Multi-Token Prediction (MTP): Equipped with three lightweight MTP modules using dense FFNs.”
“With one MTP layer for speculative decoding, the acceptance rate is about 90%.”
multimodal architecture 5
“Native multimodal in one model. V2-Pro was text-and-code only. Multimodal capability existed in the separate V2-Omni model... V2.5…”
“MiMo-V2.5 is natively omnimodal — trained from scratch to see, hear, and act across modalities”
“With native visual and audio understanding, MiMo-V2.5 reasons seamlessly across modalities”
“augmented with dedicated visual and audio encoders (both pretrained in-house) connected through lightweight projectors”
“projector warmup to align the audio and visual projectors with the language model”
context capacity 2
“supports up to 1 million tokens of context”
“during which the context window is progressively extended from 32K to 256K to 1M”
language modelling objective 1
“Training goes through five stages: text pre-training on diverse corpora”
multi-token prediction objective 1
“We support MTP in vLLM and enhance the robustness of the inference engine in the RL system.”
training precision 1
“Trained on 27T tokens using FP8 mixed precision”
data filtering 1
“applying multi-dimensional data filtering to increase reasoning pattern density in pre-training data”
data mixture & curriculum 1
“We adopt a three-stage data mixture strategy for pre-training.”
supervised fine-tuning 2
“Supervised Fine-Tuning (SFT) to build strong, foundational instruction-following skills”
“we perform RL training on a cold-started SFT model, resulting in MiMo-7B-RL”
reinforcement learning algorithm 1
“RL and MOPD”
reward modelling 2
“Each problem undergoes careful cleaning and difficulty assessment to ensure quality. We employ only rule-based accuracy rewards to avoid…”
“To mitigate the sparse reward issue for challenging code problems, we introduce a test difficulty driven code reward.”
policy distillation 1
“Multi-Teacher On-Policy Distillation (MOPD)”
rollout & RL infrastructure 3
“We develop a Seamless Rollout Engine to accelerate RL training and validation.”
“Our design integrates continuous rollout, asynchronous reward computation, and early termination to minimize GPU idle time”
“We implement a data re-sampling strategy for easy problems to enhance rollout sampling efficiency and stabilize policy updates…”
agentic post-training 2
“large-scale agentic RL”
“supervised fine-tuning and agentic post-training”
decoding strategy 3
“We support MTP in vLLM and enhance the robustness of the inference engine in the RL system.”
“--speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1”
“--enable-multi-layer-eagle”
serving parallelism 1
“--moe-a2a-backend deepep”
agent product 3
“Xiaomi explicitly positions it as a drop-in backend for scaffolds including Claude Code, OpenCode, and Kilo”
“Xiaomi explicitly positions it as a drop-in backend for scaffolds including Claude Code, OpenCode, and Kilo”
“Xiaomi explicitly positions it as a drop-in backend for scaffolds including Claude Code, OpenCode, and Kilo”
unfiled 4
“MiMo-V2.5-Pro is Xiaomi’s most capable model to date, delivering significant improvements over its predecessor, MiMo-V2-Pro, in general…”
“it makes full use of the affordances of its harness environment, manages its memory, and shapes how its own context is populated toward…”
“multimodal pre-training at scale on high-quality cross-modal data”
“Wired into an ngspice simulation loop, in about an hour of closed-loop iteration — calling the simulator, reading waveforms, tweaking…”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- MiMo-V2.5-Pro architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| mimo.xiaomi.com/mimo-v2-5 | official blog | Xiaomi MiMo | read |
| huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro | model card | Xiaomi MiMo | read |
| github.com/XiaomiMiMo/MiMo | code repo | Xiaomi MiMo | read |
| openrouter.ai/xiaomi/mimo-v2.5-pro | vendor docs | OpenRouter | read |
| artificialanalysis.ai/models/mimo-v2-5-pro | third party analysis | Artificial Analysis | read |
| marktechpost.com/2026/04/22/xiaomi-releases-mimo-v2-5-pro-and-mimo-v2-5… | news | MarkTechPost | read |
| buildfastwithai.com/blogs/xiaomi-mimo-v2-5-pro-review-2026 | third party analysis | BuildFastWithAI | read |