Open Models Almanac
ModelsMiMo-V2.5

Xiaomi

MiMo-V2.5

What MiMo-V2.5’s own documents say it is built from — every method with the sentence that describes it.

Parameters
310B / 15B active
Context
1M
Attention
sliding/global hybridhybrid sliding-window (128) + global GQA, learnable attention sink
Experts
256 routed, top-8, no shared
Input
text, image, audio, video
Licence
MIT
Released
2026-04-22

Against the consensus recipe

Its documents state 8 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 8 of the 19 architecture features checked — 3 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from the model repository’s own code (modeling_mimo_v2.py, configuration_mimo_v2.py) and config.json at revision 63651580ca, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsVerdict
Grouped-query attention in its codeadopted

Full-attention layers use 64 query heads with 4 KV heads and SWA layers use 64 query heads with 8 KV heads, both grouped via repeat_kv.

show evidencenum_attention_heads = 64num_key_value_heads = 4swa_num_key_value_heads = 8
self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
modeling_mimo_v2.py · L259
Multi-head latent attention not in its codenot stated

There is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v…

show the whole verdict

There is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v projections.

Sliding-window attention in its codeadopted

sliding_window is 128 and the hybrid_layer_pattern marks most layers as SWA (value 1), enabling create_sliding_window_causal_mask for…

show evidence

sliding_window is 128 and the hybrid_layer_pattern marks most layers as SWA (value 1), enabling create_sliding_window_causal_mask for those layers.

sliding_window = 128hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1…
self.sliding_window = getattr(config, "sliding_window", None) if is_swa else None
modeling_mimo_v2.py · L262
Interleaved sliding-window and global attention in its codeadopted

hybrid_layer_pattern interleaves full-attention layers (0) every 6 layers with sliding-window layers (1) for the rest, and the decoder…

show evidence

hybrid_layer_pattern interleaves full-attention layers (0) every 6 layers with sliding-window layers (1) for the rest, and the decoder layer sets attention_type accordingly.

hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1…
is_swa_layer = config.hybrid_layer_pattern[layer_idx] == 1
        self.attention_type = "sliding_window_attention" if is_swa_layer else "full_attention"
modeling_mimo_v2.py · L400–401
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer scoring mechanism or index_topk/index_n_heads config exists in this codebase.

Linear-attention or state-space layers alongside full attention not in its codenot stated

All decoder layers use MiMoV2Attention (softmax attention, full or sliding-window); there are no linear-attention or SSM mixer layers.

Gated DeltaNet layers not in its codenot stated

No gated delta rule / recurrent state update mechanism is implemented in the code.

Mamba-2 layers not in its codenot stated

No Mamba-2 SSD selective state-space blocks are implemented; no mamba_num_heads or ssm_state_size config exists.

Learnable attention sink in its codeadopted

SWA layers get a learned per-head attention_sink_bias parameter concatenated as an extra logit column in the softmax denominator, enabled…

show evidence

SWA layers get a learned per-head attention_sink_bias parameter concatenated as an extra logit column in the softmax denominator, enabled by add_swa_attention_sink_bias=true.

add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false
if sinks is not None:
        sinks = module.attention_sink_bias.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
        attn_weights = torch.cat([attn_weights, sinks], dim=-1)
modeling_mimo_v2.py · L89–91
self.attention_sink_bias = (
            nn.Parameter(torch.empty(self.num_attention_heads), requires_grad=False)
            if (
                (getattr(config, "add_full_attention_sink_bias", False) and not is_swa)
                or (getattr(config, "add_swa_attention_sink_bias", False) and is_swa)
            )
            else None
        )
modeling_mimo_v2.py · L268–275
Gated attention output not in its codenot stated

No sigmoid gating of attention output before o_proj is implemented; attn_output goes straight from attention interface to o_proj.

QK normalization not in its codenot stated

No RMSNorm or LayerNorm is applied to query/key states before the dot product in MiMoV2Attention; only rotary embedding is applied.

Partial RoPE code only in its codenot stated

partial_rotary_factor=0.334 makes rope_dim less than head_dim, and the attention forward splits query/key into rope and nope parts…

show evidence

partial_rotary_factor=0.334 makes rope_dim less than head_dim, and the attention forward splits query/key into rope and nope parts, applying RoPE only to the rope slice.

partial_rotary_factor = 0.334head_dim = 192
query_rope, query_nope = query_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
        key_rope, key_nope = key_states.split([self.rope_dim, self.head_dim - self.rope_dim], dim=-1)
modeling_mimo_v2.py · L306–307
YaRN RoPE scaling not in its codenot stated

rope_scaling type is 'default', not YaRN, so no YaRN scaling is applied.

show evidencerope_scaling = {"rope_type": "default", "type": "default"}
Layers without positional encoding (NoPE) not in its codenot stated

All softmax attention layers apply RoPE to the rope_dim portion of every head (partial RoPE uniformly applied); there is no per-layer…

show the whole verdict

All softmax attention layers apply RoPE to the rope_dim portion of every head (partial RoPE uniformly applied); there is no per-layer toggle disabling RoPE entirely on some attention layers.

Mixture of experts in its codeadopted

MoE layers route each token via a sigmoid-scored router (noaux_tc) to 8 of 256 routed experts per token, per moe_layer_freq.

show evidencen_routed_experts = 256num_experts_per_tok = 8moe_layer_freq = [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
self.experts = nn.ModuleList(
            [MiMoV2MLP(config, intermediate_size=config.moe_intermediate_size) for _ in range(config.n_routed_experts)]
        )
modeling_mimo_v2.py · L191–193
Shared expert not in its codenot stated

n_shared_experts is null and MiMoV2MoE has no shared-expert module in the code.

show evidencen_shared_experts = null
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The gate adds e_score_correction_bias to scores only for top-k selection (noaux_tc), while the actual weights used to combine expert…

show evidence

The gate adds e_score_correction_bias to scores only for top-k selection (noaux_tc), while the actual weights used to combine expert outputs come from the unbiased sigmoid scores.

topk_method = "noaux_tc"
scores_for_choice = scores.view(bsz * seq_len, -1) + self.e_score_correction_bias.unsqueeze(0)
modeling_mimo_v2.py · L164
_, topk_idx = torch.topk(tmp_scores, k=self.top_k, dim=-1, sorted=False)
            topk_weight = scores.gather(1, topk_idx)
modeling_mimo_v2.py · L175–176
Multi-token prediction layers code unclearadopted

There is a regex to ignore loading model.mtp.* weights, implying an MTP module may exist, but no num_nextn_predict_layers or similar…

show evidence

There is a regex to ignore loading model.mtp.* weights, implying an MTP module may exist, but no num_nextn_predict_layers or similar config key or module implementation is present in the given files.

r"model\.mtp\..*",
modeling_mimo_v2.py · L1694
SwiGLU feed-forward code only in its codenot stated

MLP and expert FFNs compute down_proj(silu(gate_proj(x)) * up_proj(x)) with hidden_act set to silu.

show evidencehidden_act = "silu"
return self.down_proj(self.act_fn(self.gate_proj(hidden_states)) * self.up_proj(hidden_states))
modeling_mimo_v2.py · L132

token mixer 8

softmax attention 7

global attention 2

Global attention core

“global attention”

Full attention used

“full-attention”

sliding window attention 3

Sliding Window Attention core

“Sliding Window Attention (SWA)”

Hybrid Sliding Window Attention core

“through its hybrid Sliding Window Attention architecture, lightweight Multi-Token Prediction, and the MOPD post-training paradigm.”

SWA-128 used

“SWA-128”

grouped-query attention 1

Grouped-query attention used

“Both SWA and GA utilize Grouped-Query Attention (GQA)”

attention sink 1

Learnable attention sink bias used

“learnable attention sink bias”

hybrid layer stacking 1

Hybrid Sliding-Window and Global Attention core

“MiMo-V2-Flash adopts a hybrid attention architecture that interleaves Sliding Window Attention (SWA) with global attention”

channel mixer 6

dense feed-forward network 1

Dense feed-forward network used

“MTP block uses dense FFN instead of MoE”

mixture of experts 5

Mixture of experts core

“MoE models suffer from inconsistent expert routing across rollout and training due to numerical precision issues”

expert routing 1

Top-k expert routing used

“it uses 256 routed experts (top-8)”

expert load balancing 3

Load balancing used

“we reference historical pass rates and, if necessary, assign new prompts to GPUs with load balancing”

Expert bias update factor used

“The expert bias update factor is set to 0.001”

Round-robin load balancing optional

“--load-balance-method round_robin”

positional encoding 1

Rotary Position Embedding used

“Rotary Positional Embedding (RoPE) is partially applied to the first 64 dimensions query and key”

prediction head 2

Multi-Token Prediction core

“We analyze the relationship between the model’s predictive uncertainty measured by next token cross-entropy and the efficiency of the…”

MTP speculative decoding used

“by repurposing MTP as a draft model for speculative decoding”

multimodal architecture 6

Native omnimodal architecture core

“MiMo-V2.5 is natively omnimodal — trained from scratch to see, hear, and act across modalities”

Native visual and audio understanding core

“With native visual and audio understanding, MiMo-V2.5 reasons seamlessly across modalities”

Vision Transformer (ViT) used

“Equipped with a 729M-param Vision Transformer (ViT) featuring hybrid window attention”

Dedicated visual and audio encoders with lightweight projectors used

“augmented with dedicated visual and audio encoders (both pretrained in-house) connected through lightweight projectors”

MiMo-Audio-initialized audio encoder used

“dedicated audio encoder initialized from the weights of MiMo-Audio”

Projector warmup used

“projector warmup to align the audio and visual projectors with the language model”

context capacity 2

1M-token context window used

“Its 1M context window supports complete documents, extended conversations, and complex task contexts in a single pass”

Progressive context extension used

“during which the context window is progressively extended from 32K to 256K to 1M”

Not used 1

MoE without shared experts not used

shared experts “and contains no shared experts.”

language modelling objective 1

Text pre-training used

“Training goes through five stages: text pre-training on diverse corpora”

multi-token prediction objective 1

Multi-Token Prediction used

“We support MTP in vLLM and enhance the robustness of the inference engine in the RL system.”

distillation objective 1

Reverse KL divergence used

“reverse KL divergence loss between student and teacher”

auxiliary loss 1

MoE sequence auxiliary loss used

“the MoE sequence auxiliary loss coefficient is set to 0.01”

optimizer 1

AdamW used

“We employ the AdamW optimizer”

learning-rate schedule 1

Cosine decay used

“cosine decay to”

training precision 1

FP8 mixed-precision training used

“we adopt an FP8 mixed-precision framework similar to DeepSeek-V3”

training stability 1

Global-norm gradient clipping used

“Gradient clipping is applied with a maximum norm of 1.0”

data filtering 1

Multi-dimensional data filtering used

“applying multi-dimensional data filtering to increase reasoning pattern density in pre-training data”

data mixture & curriculum 1

Three-stage data mixture strategy used

“We adopt a three-stage data mixture strategy for pre-training.”

supervised fine-tuning 2

Supervised fine-tuning used

“Post-training incorporates SFT, large-scale agentic RL, and Multi-Teacher On-Policy Distillation (MOPD)”

Cold-started supervised fine-tuning model used

“we perform RL training on a cold-started SFT model, resulting in MiMo-7B-RL”

reinforcement learning algorithm 3

Reinforcement learning used

“large-scale Reinforcement Learning (RL)”

Rollout Routing Replay used

“We propose Rollout Routing Replay (R3) [ma2025stabilizing] to train RL using the same routed experts from rollout”

reward modelling 3

Outcome Reward Model used

“Outcome Reward Models (ORMs)”

Rule-based verifiers used

“Each problem undergoes careful cleaning and difficulty assessment to ensure quality. We employ only rule-based accuracy rewards to avoid…”

Test-difficulty-driven code reward used

“To mitigate the sparse reward issue for challenging code problems, we introduce a test difficulty driven code reward.”

policy distillation 2

Multi-Teacher On-Policy Distillation core

“through its hybrid Sliding Window Attention architecture, lightweight Multi-Token Prediction, and the MOPD post-training paradigm.”

On-Policy Distillation used

“on-policy distillation using dense, token-level rewards from teacher logits”

rollout & RL infrastructure 8

Partial rollout used

“We integrate partial rollout [team2025kimi, fu2025areal] to partition overlong trajectories across steps”

Seamless Rollout Engine used

“we extend the Seamless Rollout Engine [xia2025mimo] and implement a Data Scheduler”

Continuous rollout with asynchronous rewards and early termination used

“Our design integrates continuous rollout, asynchronous reward computation, and early termination to minimize GPU idle time”

Data re-sampling strategy used

“We implement a data re-sampling strategy for easy problems to enhance rollout sampling efficiency and stabilize policy updates…”

Data Scheduler used

“we extend the Seamless Rollout Engine [xia2025mimo] and implement a Data Scheduler to seamlessly schedule fine-grained sequences instead…”

Staleness-aware truncated importance sampling used

“By employing staleness-aware truncated importance sampling for partial rollout, we significantly accelerate RL training without…”

Tool Manager used

“Tool Manager coordinates with Toolbox to accelerate training through environment pre-warming and sequence-level asynchronous reward…”

Toolbox used

“We implement Toolbox and Tool Manager to tackle global resource contention and local inefficiency in RL agent training.”

agentic post-training 2

Agentic reinforcement learning used

“Post-training incorporates SFT, large-scale agentic RL, and Multi-Teacher On-Policy Distillation (MOPD)”

Agentic post-training used

“supervised fine-tuning and agentic post-training”

decoding strategy 3

Multi-Token Prediction used

“We support MTP in vLLM and enhance the robustness of the inference engine in the RL system.”

Speculative decoding used

“accelerate inference via speculative decoding”

EAGLE optional

“--speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1”

KV cache management 2

Chunked prefill used

“--chunked-prefill-size 16384”

Request-level prefix cache used

“we employ a request-level prefix cache during rollout. This cache stores KVCache and MoE routed experts from prior turns”

inference quantization 1

FP8 used

“ships with native FP8 (block-wise e4m3) weights”

serving parallelism 2

DeepEP used

“--moe-a2a-backend deepep”

Attention data parallelism optional

“--enable-dp-attention”

inference kernel 1

FlashAttention-3 used

“--attention-backend fa3 --mm-attention-backend fa3”

context management 2

Discarding tool-call history used

“replicating their aggressive reset protocol, we achieve 58.3 on comparable benchmarks.”

Memory compression used

“we combat the 'Lost in the Middle' phenomenon by enforcing aggressive memory compression. When context utilization exceeds a threshold ...…”

agentic scaffolding 1

Bash commands for context retrieval used

“we adopt a Unix-inspired abstraction: tools, documents, and databases are uniformly exposed as files, enabling the model to retrieve…”

Not used 1

RadixCache not used

KV cache management “Unlike the commonly-used radix cache in current inference engines, our request-level prefix cache avoids re-prefilling or inter-request…”

agent product 3

OpenCode optional

“Xiaomi explicitly positions it as a drop-in backend for scaffolds including Claude Code, OpenCode, and Kilo”

Claude Code optional

“Xiaomi explicitly positions it as a drop-in backend for scaffolds including Claude Code, OpenCode, and Kilo”

Kilo optional

“Xiaomi explicitly positions it as a drop-in backend for scaffolds including Claude Code, OpenCode, and Kilo”

unfiled 4

agentic model core

“MiMo-V2.5-Pro is Xiaomi’s most capable model to date, delivering significant improvements over its predecessor, MiMo-V2-Pro, in general…”

harness awareness core

“it makes full use of the affordances of its harness environment, manages its memory, and shapes how its own context is populated toward…”

multimodal pre-training used

“multimodal pre-training at scale on high-quality cross-modal data”

ngspice simulation used

“Wired into an ngspice simulation loop, in about an hour of closed-loop iteration — calling the simulator, reading waveforms, tweaking…”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
mimo.xiaomi.com/mimo-v2-5official blogXiaomi MiMoread
huggingface.co/XiaomiMiMo/MiMo-V2.5model cardXiaomi MiMoread
huggingface.co/XiaomiMiMo/MiMo-V2.5-Basemodel cardXiaomi MiMoread
github.com/XiaomiMiMo/MiMocode repoXiaomi MiMoread
arxiv.org/abs/2601.02780technical reportXiaomi LLM-Coreread
recipes.vllm.ai/XiaomiMiMo/MiMo-V2.5vendor docsvLLMread
marktechpost.com/2026/04/22/xiaomi-releases-mimo-v2-5-pro-and-mimo-v2-5…newsMarkTechPostread
openrouter.ai/xiaomi/mimo-v2.5vendor docsOpenRouterread
↑