Xiaomi
MiMo-V2.6-Pro
What MiMo-V2.6-Pro’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 1.02T / 42B active
- Context
- 1M
- Attention
- sliding/global hybridhybrid sliding-window (128) + global GQA, learnable attention sink
- Experts
- 384 routed, top-8, no shared
- Input
- text, image, audio, video
- Licence
- MIT
- Released
- 2026-09-22
Against the consensus recipe
Its documents state 3 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictioncore
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningcore
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 8 of the 19 architecture features checked — 6 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from the model repository’s own code (modeling_mimo_v2.py, configuration_mimo_v2.py) and config.json at revision 73875d00b3, by anthropic/claude-sonnet-5 on 2026-09-25.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | 128 query heads share 8 key/value heads in full-attention layers (and 8 KV heads for the 128 SWA query heads too), implemented via… show evidence128 query heads share 8 key/value heads in full-attention layers (and 8 KV heads for the 128 SWA query heads too), implemented via repeat_kv grouping. num_attention_heads = 128num_key_value_heads = 8 |
| Multi-head latent attention | not in its code | not stated | The attention module projects q/k/v directly per head with no low-rank latent down/up projection (no kv_lora_rank/q_lora_rank in config or… show the whole verdictThe attention module projects q/k/v directly per head with no low-rank latent down/up projection (no kv_lora_rank/q_lora_rank in config or code). |
| Sliding-window attention code only | in its code | not stated | sliding_window is set to 128 and layers flagged 1 in hybrid_layer_pattern use is_swa attention with that window via… show evidencesliding_window is set to 128 and layers flagged 1 in hybrid_layer_pattern use is_swa attention with that window via create_sliding_window_causal_mask. sliding_window = 128hybrid_layer_pattern = [0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1… |
| Interleaved sliding-window and global attention | in its code | adopted | hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 70 layers. show evidencehybrid_layer_pattern = [0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer scoring module or top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present. |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All decoder layers use MiMoV2Attention (softmax attention, either full or sliding-window); there is no linear-attention or SSM mixer layer… show the whole verdictAll decoder layers use MiMoV2Attention (softmax attention, either full or sliding-window); there is no linear-attention or SSM mixer layer type. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta-rule recurrent module is implemented; all token mixing is standard softmax attention. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2/SSD selective state-space block is implemented anywhere in the modeling file. |
| Learnable attention sink code only | in its code | not stated | A learned per-head attention_sink_bias parameter is added as an extra logit column in the softmax denominator, enabled for both full and… show evidenceA learned per-head attention_sink_bias parameter is added as an extra logit column in the softmax denominator, enabled for both full and SWA layers via add_full_attention_sink_bias/add_swa_attention_sink_bias. add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false |
| Gated attention output | not in its code | not stated | The attention output is passed directly to o_proj with no sigmoid gating multiplication before the output projection. |
| QK normalization | not in its code | not stated | Query and key states are used directly after projection and RoPE with no RMSNorm/LayerNorm applied to them before the dot product. |
| Partial RoPE code only | in its code | not stated | partial_rotary_factor of 0.334 makes rope_dim smaller than head_dim, so RoPE is applied only to a split portion of each query/key head… show evidencepartial_rotary_factor of 0.334 makes rope_dim smaller than head_dim, so RoPE is applied only to a split portion of each query/key head with the rest (query_nope/key_nope) carrying no position signal. partial_rotary_factor = 0.334rope_parameters.partial_rotary_factor = 0.334 |
| YaRN RoPE scaling | not in its code | not stated | rope_parameters/rope_type is set to "default", not YaRN, so no YaRN scaling is applied. |
| Layers without positional encoding (NoPE) | not in its code | not stated | All attention layers (full and SWA) apply the same partial RoPE scheme uniformly; there is no layer-specific toggle disabling RoPE… show the whole verdictAll attention layers (full and SWA) apply the same partial RoPE scheme uniformly; there is no layer-specific toggle disabling RoPE entirely for some softmax attention layers. |
| Mixture of experts | in its code | adopted | MoE feed-forward layers route tokens to 8 of 384 experts per token via a learned sigmoid router (MiMoV2MoEGate), active per moe_layer_freq. show evidencen_routed_experts = 384num_experts_per_tok = 8moe_layer_freq = [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1… |
| Shared expert | not in its code | not stated | n_shared_experts is null in the config and no shared-expert module is instantiated in MiMoV2MoE. show evidencen_shared_experts = null |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method=noaux_tc) while the actual weighting… show evidenceThe MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method=noaux_tc) while the actual weighting uses the unbiased sigmoid scores. topk_method = "noaux_tc" |
| Multi-token prediction layers | code unclear | adopted | The code has a regex to ignore loading model.mtp.* weights implying an MTP module may exist, but no MTP module is defined in the given… show evidenceThe code has a regex to ignore loading model.mtp.* weights implying an MTP module may exist, but no MTP module is defined in the given file and no mtp layer count key is present in config.json. |
| SwiGLU feed-forward code only | in its code | not stated | Both dense and expert MLPs compute down_proj(act_fn(gate_proj(x)) * up_proj(x)) with hidden_act set to silu, giving SwiGLU. show evidencehidden_act = "silu" |
token mixer 4
softmax attention 2
sliding window attention 2
“Local SWA layers alternate between row-major and column-major token serialization to support information propagation along both spatial…”
“It replaces fixed, non-overlapping window attention in MiMo-VL-7B with sink-augmented SWA, enabling information exchange across window…”
hybrid layer stacking 2
“repeated hybrid blocks that interleave Local Sliding Window Attention (SWA) and Global Attention (GA)”
“MiMo-V2.6 builds a hybrid sparse MoE Transformer backbone that interleaves Local Sliding Window Attention (SWA) with Global Attention (GA)”
channel mixer 3
mixture of experts 3
“MiMo-V2.6-Pro is a Mixture of Experts (MoE) model with 1.0 trillion total parameters, but only 42 billion active parameters are used…”
“only 42 billion active parameters are used during inference.”
prediction head 1
“Multi-Token Prediction (MTP): 5-layer speculative decoder”
multimodal architecture 5
“Every four consecutive frames are grouped into an audio patch and processed by a Transformer with bidirectional self-attention confined to…”
“Audio encoding in MiMo-V2.6 consists of two stages: audio tokenization with the Audio Tokenizer, followed by patch encoding.”
“encoding runs data-parallel first—image items are balanced across ranks independently of where each sequence’s tokens land”
“In the second stage, we integrate the backbone with our in-house pre-trained ViT and audio encoder and jointly train the full model on…”
“We adopt a two-stage pre-training strategy. In the first stage, we train the language backbone on text-only data”
language modelling objective 1
“optimize the resulting VLM solely on multimodal understanding data using a cross-entropy objective”
auxiliary loss 1
“mask excludes the hit content from the loss”
filed at the root 1
“All loss computation is fused into one kernel: the policy-gradient loss (with or without top-prenormalization) and the OPD loss…”
optimizer 1
“We therefore switch to a variant of Muon optimizer, Muown, for the hidden weight matrices during mid-training”
quantization-aware training 2
“we apply quantize–dequantize (QDQ) to the experts after each parameter update.”
“We employ MXFP4 quantization-aware training (QAT) during mid-training, allowing the model to adapt to low-precision computation while…”
training stability 2
“To improve RL training stability and encourage efficient, reliable behavior, we introduce two complementary mechanisms.”
“safeguards against training drift and reward hacking.”
training parallelism 1
“the sliding-window layers exchange only the KV their queries can reach—at most a window-sized segment—so per-layer traffic is bounded by…”
training runtime 1
“Optimizer states stay in CPU memory and are copied back only for parameter updates.”
synthetic data 3
“We therefore invest in large-scale environment synthesis and curation across coding, general professional workflows, visual artifacts, and…”
“use an automated multi-agent workflow to build local software mocks that reproduce supported operations, state changes, output formats…”
“we train SFT teachers on high-quality synthetic demonstrations”
data mixture & curriculum 5
“we implement the Sample Mixer, with four mechanisms filling the specified training distribution”
“we train the MiMo-V2.6 model series on a diverse, agent-centric data mixture”
“we have maintained a Data Scheduler that targets a specified training distribution across data sources”
“we have maintained a Data Scheduler that targets a specified training distribution across data sources, together with dynamic sampler and…”
“These weights drive smooth weighted round-robin”
sequence packing 1
“one packer per training tensor-parallel (TP) group serves every rank in the group”
supervised fine-tuning 3
“this checkpoint is an SFT starting point for RL research, not a finished assistant”
“Cold start from self-correction — the model reflects on and rewrites its own misaligned turns into grounded next steps.”
“We obtain the model by supervised fine-tuning Qwen3.5-9B (Qwen Team, 2025) on MiMo-generated data spanning coding, general-domain, visual…”
reinforcement learning algorithm 17
“a configurable Penalty Module applies loss masking and advantage shaping”
“groups whose advantages are all zero are dropped by default”
“Fully asynchronous Group Relative Policy Optimization (GRPO) on very large batches”
“Through our proposed Groupwise Reward Synthesis (GRS) and Groupwise Advantage Redistribution (GAR), these fine-grained distinctions are…”
“we incorporate a dynamic sampler to filter groups that are all-pass or all-fail”
“Rollout Routing Replay (R3) (Ma et al., 2025) records the expert indices used during rollout and replays them during training, reproducing…”
“advantage shaping sets, scales, or subtracts the advantages on hit tokens”
“we conduct GRPO training separately for coding, cybersecurity, general-domain, and visual tasks using the corresponding released…”
“This masks flagged tokens in positive trajectories and penalizes them more strongly in negative trajectories.”
“We therefore freeze the router for RL training; the frozen-router run keeps all three statistics flat”
“For each prompt q with G sampled rollouts, let R_i be the original outcome reward, P_q the indices of successful rollouts, and ℓ_i = |o_i|…”
“a context with no surviving model turns is dropped, a sequence with no surviving context receives zero advantage, and a sample with no…”
“quality factors first downweight lower-quality passes, after which a common factor redistributes the removed positive advantage mass among…”
“we record each token’s candidate set during rollout and renormalize the training log-probabilities within it, matching the sampler’s…”
“One mixed RL run across coding, general agents, visual, and cybersecurity — not separate per-domain runs.”
“The coding results in Table 6 are obtained through single-harness RL.”
“When evidence confirms dependence on an external or leaked answer, we reset the trajectory’s effective reward to zero and treat it as a…”
reward modelling 13
“we introduce groupwise agentic grading to provide more informative reward signals beyond binary test cases”
“A dedicated hack agent then probes the prepared environments for remaining leaks and exploitable weaknesses.”
“Through our proposed Groupwise Reward Synthesis (GRS) and Groupwise Advantage Redistribution (GAR), these fine-grained distinctions are…”
“Throughout RL, environment hardening, adversarial screening, and verifier cross-checks keep the loop honest against reward hacking.”
“Behavior rubrics describe observable practices that support reliable problem solving, such as gathering relevant evidence and checking the…”
“comparing passing patches along five dimensions: the suitability of the solution approach, precision in implementing that approach without…”
“monitor records metrics alone”
“We synthesize the final training reward by multiplying this test reward by the two rubric scores”
“To test resistance to reward hacking, we include negative checks for unintended changes to unrelated files or databases”
“We regularly audit agent trajectories offline for reward hacking throughout training.”
“A Rule judges a segment, context, or sequence by handcrafted logic or a model-based judge.”
“motivating segment-level rules for format violations and tool call errors such as malformed markup, invalid tool names, or malformed…”
“Solution rubrics describe task-relevant properties of a good implementation, including satisfaction of the requirements, appropriate…”
policy distillation 6
“After mixed RL, we use Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD2) to combine capabilities from teachers trained for…”
“SFT data provide prefix contexts rather than fixed continuation targets.”
“we retain autonomous student rollouts in domains with suitable mixRL teachers”
“a Qwen3.5-9B fine-tune trained on MiMo-generated data”
“These progressive qualitative improvements align with the quantitative trajectory (61.7 → 64.0 → 72.4) observed in Table 6, jointly…”
“Standard MOPD uses RL teachers to supervise full student rollouts.”
rollout & RL infrastructure 26
“we scale batch size and training throughput through fully asynchronous training, processing thousands of long-horizon rollouts and…”
“each sequence runs as an Agent Loop that owns the environment lifecycle, manages the dialogues it produces, and calls the inference engine…”
“we adopt an agent-centric execution model and organize trajectory data as a hierarchy”
“We organize the trajectory data into a four-level hierarchy: Sample → Sequence → Context → Segment.”
“the Payload Porter keeps the driver scheduling on lightweight metadata alone. Heavy payloads are written once to a distributed store and…”
“the Harness Pool hosts concurrent harness instances and Agent Loops in persistent multi-tenant actor pools”
“We jointly estimate KV demand and expected inference concurrency to guide the admission and placement of new rollouts across ranks.”
“we have maintained a Data Scheduler that targets a specified training distribution across data sources, together with dynamic sampler and…”
“Slower sources need more concurrent rollouts to sustain the same training contribution.”
“each rollout runs in an isolated sandbox that can be restored to a fixed initial state”
“Within these budgets, scheduling balances long-run generation demand with progress toward the current training batch.”
“blocking work—environment operations and tokenization—runs on background threads”
“The composite early stop strategy halts the rollout as soon as its rule fires, zeroes the outcome reward, applies separate actions to the…”
“decouple the control plane from the data plane to buffer and transfer massive trajectories carrying routing and multi-modal payloads”
“Prefix matching locates the dialogue that an incoming request extends, and only the new suffix is tokenized and passed to the inference…”
“We assign each source a scheduling budget of (1 + pᵢ)mᵢ groups, where the oversampling ratio pᵢ satisfies”
“We configure harness codebases, agent behavior, and environment settings separately to accommodate diverse tasks within a single training…”
“During rollout, the Agent Loop therefore ships only the multi-modal delta between requests”
“These advances are supported by asynchronous training, mixed-task rollout infrastructure, training–inference consistency mechanisms”
“Tasks and multiple harnesses are mixed in the same batch”
“Multi-modal payloads follow the same meta/payload split, but demand extra care”
“Per-source priors estimate a rollout’s total sequence length (input and generated tokens), summed across contexts”
“A sample mixing mechanism, working together with dynamic sampling and partial rollout, stabilizes per-task sample composition in train…”
“We use sample replay to accelerate initial collection from slow sources.”
“These advances are supported by asynchronous training, mixed-task rollout infrastructure, training–inference consistency mechanisms”
“A group-wise grader, when configured, runs fully asynchronously alongside the Agent Loops”
agentic post-training 7
“a practical approach to advancing foundation models through large-scale agentic reinforcement learning.”
“Throughout RL, environment hardening, adversarial screening, and verifier cross-checks keep the loop honest against reward hacking.”
“We enforce container-level network isolation to restrict access to upstream fixes and alternative package versions that may contain the…”
“We therefore remove build logs, verifier outputs, residual patches, and project-generated binaries or bytecode that could expose it.”
“MiMo-V2.6 breaks the task into sub-tasks and coordinates multiple agents to build 3D scenes, interaction logic, and visual checks”
“we jointly optimize the model across four mini-harnesses and evaluate it on these harnesses and three additional held-out harnesses.”
“we design a unified trajectory representation and a penalty mechanism that together refine the learning signal”
mid-training & continual pretraining 2
“Building on an omni-capable foundation and agent-centric mid-training, we scale RL along three dimensions”
“For each case, MiMo reflects on the faulty reasoning, revises the relevant turn, and continues with actions grounded in the task…”
1 more mentioned or evaluated
rollout & RL infrastructure “Deficit-corrected scheduling (α = 0.5) improves occupancy stability over Deficit-based scheduling (α = 0) and collection balance over…”
decoding strategy 4
“Speculative decoding in MiMo-V2.6 uses a multi-token prediction (MTP) module following the block diffusion design of DFlash”
“it predicts 7 subsequent tokens in a single forward pass for parallel verification by the backbone”
“DFlash is then finetuned on early RL rollout logs resampled to match the RL training distribution.”
“At large RL batch sizes, we select the draft block size based on end-to-end throughput rather than acceptance rate alone.”
reasoning control 1
“It uses extended thinking or chain-of-thought reasoning to work through complex problems before providing an answer.”
KV cache management 3
“each dialogue context has a persistent key: within one policy version, later turns hit the cached KV—including generated tokens—and…”
“Offloading and restoration run on side CUDA streams rather than the compute stream, never stalling generation.”
“a state resides in HBM during GPU time and in a pinned host pool during tool time.”
inference quantization 2
“quant_method: fp8 with e4m3 and dynamic activation scaling”
“Low-precision draft computation (FP8) is also applied to reduce drafting overhead.”
serving parallelism 1
“its SGLang example spans two nodes with --tp 16 --ep 16 and DeepEP for expert parallelism”
inference scheduling 1
“a greedy heuristic selects the feasible rank with the greatest remaining capacity”
agentic scaffolding 2
“All mini-harnesses start from the same minimal agent loop, which already includes the components needed for task completion—system prompt…”
“If you put Pro behind an agent loop, set a request timeout and a retry.”
inference engine 3
“Xiaomi’s model card serves it with vLLM at --tensor-parallel-size 8”
“Follow the vLLM MiMo-V2.5 recipe.”
“For best performance, follow the SGLang MiMo cookbook.”
filed at the root 3
“scripts were run against stubbed pg_dump and aws binaries with a missing-variable test, a real run, and a retention check using fake…”
“we further assess their alignment through rollout-based auditing. Each task is attempted four times by a coding agent.”
“build 3D scenes, interaction logic, and visual checks—refining renders until a playable world is complete”
evaluation harness 1
“Task completion is evaluated using atomic, binary rubric items: code-based checks verify deterministic properties”
human & real-world evaluation 1
“a same-prompt DevOps test graded by kubeconform, shellcheck, tofu validate, a live k3s cluster, and a behaviour test for every bash script”
other 3
filed at the root 3
“Every artifact went through its linter, then the part that matters: manifests were applied to k3s and curled through the Service, and…”
“turning concepts like Fourier decomposition and convex hulls into clear, intuitive animations”
“We align the training and inference engines on MoE routing and top-p sampling candidate sets”
unfiled 10
“The first Transformer block uses global attention with a dense FFN.”
“store_dtype: mxfp4 with a block size of 32”
“refining renders until a playable world is complete”
“more than 7,000 RL task environments”
“reviewing literature, forming hypotheses, simulating binding strength, and shortlisting top candidates”
“the final RL stage ran live for under six days”
“We pinned OpenRouter to the first-party Xiaomi endpoint with fallbacks disabled, so every call below hit Xiaomi’s servers and not a…”
“We check reward stability through repeated execution.”
“Run ShellCheck at info level on anything a model writes.”
“The fix is to give nginx writable emptyDir volumes at /var/cache/nginx and /var/run, which is what Pro’s first run did.”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| mimo.xiaomi.com/mimo-v2-6 | official blog | Xiaomi MiMo | not fetched |
| huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/resolve/main/MiMo_V2_6_techn… | technical report | Xiaomi MiMo | read |
| huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL | model card | Xiaomi MiMo | read |
| mimo.mi.com/models/en-US/mimo-v2.6-pro | vendor docs | Xiaomi MiMo | read |
| openrouter.ai/xiaomi/mimo-v2.6-pro | vendor docs | OpenRouter | read |
| artificialanalysis.ai/models/mimo-v2-6-pro | third party analysis | Artificial Analysis | read |
| computingforgeeks.com/xiaomi-mimo-v2-6-pro-flash | third party analysis | ComputingForGeeks | read |
| venturebeat.com/technology/better-than-deepseek-xiaomis-mimo-v2-6-pro-d… | news | VentureBeat | not fetched |
| arxiv.org/abs/2602.06036 | technical report | Z Lab (Chen et al.) | not fetched |