Xiaomi · usage rank #10
MiMo-V2.6-Flash
What MiMo-V2.6-Flash’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 309B / 15B active
- Context
- 1M
- Attention
- sliding/global hybrid39 SWA (window 128) : 9 global GQA layers, attention sink
- Experts
- 256 routed, top-8, no shared
- Input
- text, image, audio, video
- Licence
- MIT
- Released
- 2026-09-22
Rank by open-weight tokens on OpenRouter, week of 2026-10-26.
Against the consensus recipe
Its documents state 3 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictioncore
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningcore
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 9 of the 19 architecture features checked — 6 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from the model repository’s own code (modeling_mimo_v2.py, configuration_mimo_v2.py) and config.json at revision 5711b26816, by anthropic/claude-sonnet-5 on 2026-09-25.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | Full-attention layers use 64 query heads with 4 KV heads (and SWA layers use 64 query heads with 8 KV heads), repeated via repeat_kv… show evidenceFull-attention layers use 64 query heads with 4 KV heads (and SWA layers use 64 query heads with 8 KV heads), repeated via repeat_kv, giving grouped-query attention throughout. num_attention_heads = 64num_key_value_heads = 4swa_num_key_value_heads = 8 |
| Multi-head latent attention | not in its code | not stated | There is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v… show the whole verdictThere is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v projections with GQA instead. |
| Sliding-window attention code only | in its code | not stated | Layers flagged 1 in hybrid_layer_pattern are is_swa layers whose sliding_window is set to the non-null config value 128, and a… show evidenceLayers flagged 1 in hybrid_layer_pattern are is_swa layers whose sliding_window is set to the non-null config value 128, and a sliding-window causal mask is built for them. sliding_window = 128hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1… |
| Interleaved sliding-window and global attention | in its code | adopted | hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 48… show evidencehybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 48 decoder layers. hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer scoring or top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present. |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All token-mixing layers in this file are softmax attention (full or sliding-window); there is no linear-attention, Mamba, or state-space… show the whole verdictAll token-mixing layers in this file are softmax attention (full or sliding-window); there is no linear-attention, Mamba, or state-space mixer implementation. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta-rule recurrent mixer is implemented; only softmax attention layers exist. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2/SSD selective state-space block is implemented anywhere in the given files. |
| Learnable attention sink code only | in its code | not stated | Language-model attention has a learned per-head attention_sink_bias parameter appended as an extra logit column in eager attention… show evidenceLanguage-model attention has a learned per-head attention_sink_bias parameter appended as an extra logit column in eager attention softmax, enabled for SWA layers by add_swa_attention_sink_bias=true. add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false |
| Gated attention output | not in its code | not stated | The language-model attention output goes straight through o_proj with no sigmoid gate multiplying it; no attn_output_gate/use_output_gate… show the whole verdictThe language-model attention output goes straight through o_proj with no sigmoid gate multiplying it; no attn_output_gate/use_output_gate config or code exists for the LM. |
| QK normalization | not in its code | not stated | Query and key tensors are used directly for RoPE and attention without any RMSNorm/LayerNorm normalization step in MiMoV2Attention. |
| Partial RoPE code only | in its code | not stated | Each head's dimension is split into a rotary part (rope_dim, sized by partial_rotary_factor 0.334) and a non-rotary part (head_dim… show evidenceEach head's dimension is split into a rotary part (rope_dim, sized by partial_rotary_factor 0.334) and a non-rotary part (head_dim - rope_dim) that carries no position signal. partial_rotary_factor = 0.334head_dim = 192 |
| YaRN RoPE scaling | not in its code | not stated | rope_parameters/rope_type is set to "default" in this config, not YaRN, so no YaRN scaling is applied. |
| Layers without positional encoding (NoPE) | not in its code | not stated | Every attention layer (full or sliding-window) applies rotary embeddings via its own rotary embedding module; there is no layer that skips… show the whole verdictEvery attention layer (full or sliding-window) applies rotary embeddings via its own rotary embedding module; there is no layer that skips RoPE entirely for softmax attention. |
| Mixture of experts | in its code | adopted | MoE feed-forward layers route each token via a sigmoid-scored top-8-of-256 router (MiMoV2MoEGate/MiMoV2MoE) applied on layers flagged in… show evidenceMoE feed-forward layers route each token via a sigmoid-scored top-8-of-256 router (MiMoV2MoEGate/MiMoV2MoE) applied on layers flagged in moe_layer_freq. n_routed_experts = 256num_experts_per_tok = 8 |
| Shared expert | not in its code | not stated | n_shared_experts is null in this config, and MiMoV2MoE builds no additional always-on expert module. show evidencen_shared_experts = null |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds e_score_correction_bias to scores only for top-k selection (noaux_tc topk_method), while topk_weight for weighting is… show evidenceThe router adds e_score_correction_bias to scores only for top-k selection (noaux_tc topk_method), while topk_weight for weighting is gathered from the unbiased sigmoid scores. topk_method = "noaux_tc" |
| Multi-token prediction layers | in its code | adopted | config declares num_nextn_predict_layers=3 for MTP modules, even though the given modeling file does not build them (weights are ignored… show evidenceconfig declares num_nextn_predict_layers=3 for MTP modules, even though the given modeling file does not build them (weights are ignored via the mtp key pattern), which counts per the check's rule for config-declared nonzero MTP counts. num_nextn_predict_layers = 3 |
| SwiGLU feed-forward code only | in its code | not stated | MiMoV2MLP (used both as dense MLP and MoE expert) computes down_proj(act_fn(gate_proj(x)) * up_proj(x)) with hidden_act=silu, i.e. SwiGLU. show evidencehidden_act = "silu" |
token mixer 4
softmax attention 2
sliding window attention 2
“Local SWA layers alternate between row-major and column-major token serialization to support information propagation along both spatial…”
“It replaces fixed, non-overlapping window attention in MiMo-VL-7B with sink-augmented SWA, enabling information exchange across window…”
hybrid layer stacking 2
“repeated hybrid blocks that interleave Local Sliding Window Attention (SWA) and Global Attention (GA)”
“MiMo-V2.6 builds a hybrid sparse MoE Transformer backbone that interleaves Local Sliding Window Attention (SWA) with Global Attention (GA)”
channel mixer 2
mixture of experts 2
“Built on a Mixture-of-Experts architecture with 309B total parameters and 15B activated per token”
prediction head 1
“Multi-Token Prediction (MTP): 5-layer speculative decoder”
multimodal architecture 6
“Every four consecutive frames are grouped into an audio patch and processed by a Transformer with bidirectional self-attention confined to…”
“Joint input and understanding of images, video, audio, and text”
“Audio encoding in MiMo-V2.6 consists of two stages: audio tokenization with the Audio Tokenizer, followed by patch encoding.”
“encoding runs data-parallel first—image items are balanced across ranks independently of where each sequence’s tokens land”
“In the second stage, we integrate the backbone with our in-house pre-trained ViT and audio encoder and jointly train the full model on…”
“We adopt a two-stage pre-training strategy. In the first stage, we train the language backbone on text-only data”
context capacity 1
“1M-token context window, suited for mixed-modality input and multi-agent collaboration”
language modelling objective 1
“optimize the resulting VLM solely on multimodal understanding data using a cross-entropy objective”
auxiliary loss 1
“mask excludes the hit content from the loss”
filed at the root 1
“All loss computation is fused into one kernel: the policy-gradient loss (with or without top-prenormalization) and the OPD loss…”
optimizer 1
“We therefore switch to a variant of Muon optimizer, Muown, for the hidden weight matrices during mid-training”
quantization-aware training 2
“we apply quantize–dequantize (QDQ) to the experts after each parameter update.”
“We employ MXFP4 quantization-aware training (QAT) during mid-training, allowing the model to adapt to low-precision computation while…”
training stability 2
“To improve RL training stability and encourage efficient, reliable behavior, we introduce two complementary mechanisms.”
“safeguards against training drift and reward hacking.”
training parallelism 1
“the sliding-window layers exchange only the KV their queries can reach—at most a window-sized segment—so per-layer traffic is bounded by…”
training runtime 1
“Optimizer states stay in CPU memory and are copied back only for parameter updates.”
synthetic data 3
“We therefore invest in large-scale environment synthesis and curation across coding, general professional workflows, visual artifacts, and…”
“use an automated multi-agent workflow to build local software mocks that reproduce supported operations, state changes, output formats…”
“we train SFT teachers on high-quality synthetic demonstrations”
data mixture & curriculum 5
“we implement the Sample Mixer, with four mechanisms filling the specified training distribution”
“we train the MiMo-V2.6 model series on a diverse, agent-centric data mixture”
“we have maintained a Data Scheduler that targets a specified training distribution across data sources”
“we have maintained a Data Scheduler that targets a specified training distribution across data sources, together with dynamic sampler and…”
“These weights drive smooth weighted round-robin”
sequence packing 1
“one packer per training tensor-parallel (TP) group serves every rank in the group”
supervised fine-tuning 3
“this checkpoint is an SFT starting point for RL research, not a finished assistant”
“Cold start from self-correction — the model reflects on and rewrites its own misaligned turns into grounded next steps.”
“We obtain the model by supervised fine-tuning Qwen3.5-9B (Qwen Team, 2025) on MiMo-generated data spanning coding, general-domain, visual…”
reinforcement learning algorithm 16
“Fully asynchronous Group Relative Policy Optimization (GRPO) on very large batches”
“Groupwise Advantage Redistribution (GAR) ranks passing trajectories online and moves advantage toward higher-quality solutions.”
“a configurable Penalty Module applies loss masking and advantage shaping”
“groups whose advantages are all zero are dropped by default”
“we incorporate a dynamic sampler to filter groups that are all-pass or all-fail”
“Rollout Routing Replay (R3) (Ma et al., 2025) records the expert indices used during rollout and replays them during training, reproducing…”
“advantage shaping sets, scales, or subtracts the advantages on hit tokens”
“we conduct GRPO training separately for coding, cybersecurity, general-domain, and visual tasks using the corresponding released…”
“This masks flagged tokens in positive trajectories and penalizes them more strongly in negative trajectories.”
“We therefore freeze the router for RL training; the frozen-router run keeps all three statistics flat”
“For each prompt q with G sampled rollouts, let R_i be the original outcome reward, P_q the indices of successful rollouts, and ℓ_i = |o_i|…”
“a context with no surviving model turns is dropped, a sequence with no surviving context receives zero advantage, and a sample with no…”
“quality factors first downweight lower-quality passes, after which a common factor redistributes the removed positive advantage mass among…”
“we record each token’s candidate set during rollout and renormalize the training log-probabilities within it, matching the sampler’s…”
“The coding results in Table 6 are obtained through single-harness RL.”
“When evidence confirms dependence on an external or leaked answer, we reset the trajectory’s effective reward to zero and treat it as a…”
reward modelling 13
“we introduce groupwise agentic grading to provide more informative reward signals beyond binary test cases”
“Groupwise Reward Synthesis (GRS) builds task-specific rubrics offline from contrasting rollouts and fuses rubric quality with test outcomes”
“A dedicated hack agent then probes the prepared environments for remaining leaks and exploitable weaknesses.”
“Throughout RL, environment hardening, adversarial screening, and verifier cross-checks keep the loop honest against reward hacking.”
“Behavior rubrics describe observable practices that support reliable problem solving, such as gathering relevant evidence and checking the…”
“comparing passing patches along five dimensions: the suitability of the solution approach, precision in implementing that approach without…”
“monitor records metrics alone”
“We synthesize the final training reward by multiplying this test reward by the two rubric scores”
“To test resistance to reward hacking, we include negative checks for unintended changes to unrelated files or databases”
“We regularly audit agent trajectories offline for reward hacking throughout training.”
“A Rule judges a segment, context, or sequence by handcrafted logic or a model-based judge.”
“motivating segment-level rules for format violations and tool call errors such as malformed markup, invalid tool names, or malformed…”
“Solution rubrics describe task-relevant properties of a good implementation, including satisfaction of the requirements, appropriate…”
policy distillation 6
“After mixed RL, we use Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD2) to combine capabilities from teachers trained for…”
“SFT data provide prefix contexts rather than fixed continuation targets.”
“we retain autonomous student rollouts in domains with suitable mixRL teachers”
“a Qwen3.5-9B fine-tune trained on MiMo-generated data”
“These progressive qualitative improvements align with the quantitative trajectory (61.7 → 64.0 → 72.4) observed in Table 6, jointly…”
“Standard MOPD uses RL teachers to supervise full student rollouts.”
rollout & RL infrastructure 25
“we scale batch size and training throughput through fully asynchronous training, processing thousands of long-horizon rollouts and…”
“each sequence runs as an Agent Loop that owns the environment lifecycle, manages the dialogues it produces, and calls the inference engine…”
“we adopt an agent-centric execution model and organize trajectory data as a hierarchy”
“We organize the trajectory data into a four-level hierarchy: Sample → Sequence → Context → Segment.”
“the Payload Porter keeps the driver scheduling on lightweight metadata alone. Heavy payloads are written once to a distributed store and…”
“the Harness Pool hosts concurrent harness instances and Agent Loops in persistent multi-tenant actor pools”
“We jointly estimate KV demand and expected inference concurrency to guide the admission and placement of new rollouts across ranks.”
“we have maintained a Data Scheduler that targets a specified training distribution across data sources, together with dynamic sampler and…”
“Slower sources need more concurrent rollouts to sustain the same training contribution.”
“each rollout runs in an isolated sandbox that can be restored to a fixed initial state”
“Within these budgets, scheduling balances long-run generation demand with progress toward the current training batch.”
“blocking work—environment operations and tokenization—runs on background threads”
“The composite early stop strategy halts the rollout as soon as its rule fires, zeroes the outcome reward, applies separate actions to the…”
“decouple the control plane from the data plane to buffer and transfer massive trajectories carrying routing and multi-modal payloads”
“Prefix matching locates the dialogue that an incoming request extends, and only the new suffix is tokenized and passed to the inference…”
“We assign each source a scheduling budget of (1 + pᵢ)mᵢ groups, where the oversampling ratio pᵢ satisfies”
“We configure harness codebases, agent behavior, and environment settings separately to accommodate diverse tasks within a single training…”
“During rollout, the Agent Loop therefore ships only the multi-modal delta between requests”
“These advances are supported by asynchronous training, mixed-task rollout infrastructure, training–inference consistency mechanisms”
“Multi-modal payloads follow the same meta/payload split, but demand extra care”
“Per-source priors estimate a rollout’s total sequence length (input and generated tokens), summed across contexts”
“A sample mixing mechanism, working together with dynamic sampling and partial rollout, stabilizes per-task sample composition in train…”
“We use sample replay to accelerate initial collection from slow sources.”
“These advances are supported by asynchronous training, mixed-task rollout infrastructure, training–inference consistency mechanisms”
“A group-wise grader, when configured, runs fully asynchronously alongside the Agent Loops”
agentic post-training 6
“a practical approach to advancing foundation models through large-scale agentic reinforcement learning.”
“Throughout RL, environment hardening, adversarial screening, and verifier cross-checks keep the loop honest against reward hacking.”
“We enforce container-level network isolation to restrict access to upstream fixes and alternative package versions that may contain the…”
“We therefore remove build logs, verifier outputs, residual patches, and project-generated binaries or bytecode that could expose it.”
“we jointly optimize the model across four mini-harnesses and evaluate it on these harnesses and three additional held-out harnesses.”
“we design a unified trajectory representation and a penalty mechanism that together refine the learning signal”
mid-training & continual pretraining 2
“Building on an omni-capable foundation and agent-centric mid-training, we scale RL along three dimensions”
“For each case, MiMo reflects on the faulty reasoning, revises the relevant turn, and continues with actions grounded in the task…”
1 more mentioned or evaluated
rollout & RL infrastructure “Deficit-corrected scheduling (α = 0.5) improves occupancy stability over Deficit-based scheduling (α = 0) and collection balance over…”
decoding strategy 6
“5-layer SWA MTP drafter (DFlash-style). Predicts 7 subsequent tokens per forward pass for parallel verification.”
“it predicts 7 subsequent tokens in a single forward pass for parallel verification by the backbone”
“DFlash is then finetuned on early RL rollout logs resampled to match the RL training distribution.”
“At large RL batch sizes, we select the draft block size based on end-to-end throughput rather than acceptance rate alone.”
“--speculative-algorithm EAGLE”
“--speculative-algorithm EAGLE”
KV cache management 3
“each dialogue context has a persistent key: within one policy version, later turns hit the cached KV—including generated tokens—and…”
“Offloading and restoration run on side CUDA streams rather than the compute stream, never stalling generation.”
“a state resides in HBM during GPU time and in a pinned host pool during tool time.”
inference quantization 2
“quant_method: fp8 with e4m3 and dynamic activation scaling”
“Low-precision draft computation (FP8) is also applied to reduce drafting overhead.”
serving parallelism 1
“its SGLang example spans two nodes with --tp 16 --ep 16 and DeepEP for expert parallelism”
inference scheduling 1
“a greedy heuristic selects the feasible rank with the greatest remaining capacity”
agentic scaffolding 2
“All mini-harnesses start from the same minimal agent loop, which already includes the components needed for task completion—system prompt…”
“If you put Pro behind an agent loop, set a request timeout and a retry.”
1 more mentioned or evaluated
agentic scaffolding “suited for mixed-modality input and multi-agent collaboration”
inference engine 1
“Xiaomi’s model card serves it with vLLM at --tensor-parallel-size 8”
filed at the root 2
“scripts were run against stubbed pg_dump and aws binaries with a missing-variable test, a real run, and a retention check using fake…”
“we further assess their alignment through rollout-based auditing. Each task is attempted four times by a coding agent.”
evaluation harness 1
“Task completion is evaluated using atomic, binary rubric items: code-based checks verify deterministic properties”
human & real-world evaluation 1
“a same-prompt DevOps test graded by kubeconform, shellcheck, tofu validate, a live k3s cluster, and a behaviour test for every bash script”
other 2
filed at the root 2
“Every artifact went through its linter, then the part that matters: manifests were applied to k3s and curled through the Service, and…”
“We align the training and inference engines on MoE routing and top-p sampling candidate sets”
unfiled 8
“store_dtype: mxfp4 with a block size of 32”
“more than 7,000 RL task environments”
“the final RL stage ran live for under six days”
“Optimized on real harness environments—drop-in ready for agent harnesses like MiMo Desktop”
“We pinned OpenRouter to the first-party Xiaomi endpoint with fallbacks disabled, so every call below hit Xiaomi’s servers and not a…”
“We check reward stability through repeated execution.”
“Run ShellCheck at info level on anything a model writes.”
“The fix is to give nginx writable emptyDir volumes at /var/cache/nginx and /var/run, which is what Pro’s first run did.”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| mimo.xiaomi.com/mimo-v2-6 | official blog | Xiaomi MiMo | not fetched |
| huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/resolve/main/MiMo_V2_6_techn… | technical report | Xiaomi MiMo | read |
| huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL | model card | Xiaomi MiMo | read |
| mimo.mi.com/models/en-US/mimo-v2.6-flash | vendor docs | Xiaomi MiMo | read |
| openrouter.ai/xiaomi/mimo-v2.6-flash | vendor docs | OpenRouter | read |
| computingforgeeks.com/xiaomi-mimo-v2-6-pro-flash | third party analysis | ComputingForGeeks | read |