MiniMax
MiniMax-M3
What MiniMax-M3’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- ~428B / ~23B active
- Context
- 1M
- Attention
- sparse (indexer)MiniMax Sparse Attention (MSA) with GQA
- Experts
- 128 routed + 1 shared, top-4
- Input
- text, image, video
- Licence
- MiniMax Community License
- Released
- 2026-06-01
Against the consensus recipe
Its documents state 3 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentioncore
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modeoptional
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 9 of the 19 architecture features checked — 6 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_minimax_m3_vl.py, configuration_minimax_m3_vl.py) and config.json at revision d95e30cb91, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | in its code | adopted | The language model uses 64 query heads sharing 4 key/value heads via repeat_kv grouping. show evidencetext_config.num_attention_heads = 64text_config.num_key_value_heads = 4 |
| Multi-head latent attention | not in its code | not used | Attention uses standard per-head q/k/v projections with GQA, with no latent low-rank KV projection or lora rank config present. |
| Sliding-window attention | not in its code | discussed | No sliding_window config key or windowed masking code is present; layer_types only distinguishes full_attention from the sparse indexer… show the whole verdictNo sliding_window config key or windowed masking code is present; layer_types only distinguishes full_attention from the sparse indexer layers. |
| Interleaved sliding-window and global attention | not in its code | not stated | layer_types alternates only between full_attention and the lightning-indexer sparse attention, not sliding-window vs global attention. |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) code only | in its code | discussed | A lightning indexer scores past tokens in blocks and selects top-k blocks per query, enabled on most layers via… show evidenceA lightning indexer scores past tokens in blocks and selects top-k blocks per query, enabled on most layers via sparse_attention_freq/layer_types. text_config.sparse_attention_config.use_sparse_attention = truetext_config.sparse_attention_config.sparse_attention_freq = [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1… |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | There is no recurrent/state-space linear-attention layer type; only softmax attention (full or sparse-indexer-selected) layers exist. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta rule recurrent mixer or related config keys (linear_num_key_heads, linear_conv_kernel_dim) appear in this model. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 SSM blocks or related config keys (mamba_num_heads, ssm_state_size) are present. |
| Learnable attention sink | not in its code | not stated | The attention softmax has no extra learned sink logit; eager_attention_forward only uses the standard query/key/value matmuls. |
| Gated attention output | not in its code | not stated | attention_output_gate is set false in text_config and no sigmoid gating of attention output is implemented before o_proj. show evidencetext_config.attention_output_gate = false |
| QK normalization code only | in its code | not stated | use_qk_norm is true and the attention module applies per-head RMSNorm to queries and keys before RoPE/dot-product. show evidencetext_config.use_qk_norm = truetext_config.qk_norm_type = "per_head" |
| Partial RoPE code only | in its code | not stated | partial_rotary_factor is 0.5 and apply_rotary_pos_emb only rotates the first rotary_dim channels, passing the rest through unrotated. show evidencetext_config.partial_rotary_factor = 0.5text_config.rotary_dim = 64 |
| YaRN RoPE scaling | code unclear | not stated | rope_parameters is not given a literal value in the provided config.json (it is None in the class default) so whether YaRN scaling is… show evidencerope_parameters is not given a literal value in the provided config.json (it is None in the class default) so whether YaRN scaling is active cannot be determined from these files. |
| Layers without positional encoding (NoPE) | not in its code | not stated | All attention layers apply the same rotary embedding via apply_rotary_pos_emb; there is no per-layer toggle to skip positional encoding… show the whole verdictAll attention layers apply the same rotary embedding via apply_rotary_pos_emb; there is no per-layer toggle to skip positional encoding entirely. |
| Mixture of experts | in its code | adopted | MoE feed-forward layers route each token to 4 of 128 experts via a learned sigmoid-scored router, enabled on most layers via… show evidenceMoE feed-forward layers route each token to 4 of 128 experts via a learned sigmoid-scored router, enabled on most layers via mlp_layer_types. text_config.num_local_experts = 128text_config.num_experts_per_tok = 4 |
| Shared expert | in its code | adopted | Each MoE block runs a dense shared expert of shared_intermediate_size=3072 (n_shared_experts=1) whose output is added to the routed… show evidenceEach MoE block runs a dense shared expert of shared_intermediate_size=3072 (n_shared_experts=1) whose output is added to the routed experts' output. text_config.n_shared_experts = 1text_config.shared_intermediate_size = 3072 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds e_score_correction_bias to the sigmoid scores solely to pick top-k experts, while actual routing weights are taken from… show evidenceThe router adds e_score_correction_bias to the sigmoid scores solely to pick top-k experts, while actual routing weights are taken from the unbiased scores. text_config.use_routing_bias = true |
| Multi-token prediction layers code only | in its code | not stated | The config declares num_mtp_modules=7 and num_nextn_predict_layers=1, and the model explicitly ignores mtp.* weights on load, confirming… show evidenceThe config declares num_mtp_modules=7 and num_nextn_predict_layers=1, and the model explicitly ignores mtp.* weights on load, confirming MTP modules are part of the checkpoint. text_config.num_mtp_modules = 7text_config.num_nextn_predict_layers = 1 |
| SwiGLU feed-forward code only | in its code | not stated | Dense and expert MLPs compute a SiLU-sigmoid gated SwiGLU-OAI variant (gate * sigmoid(gate*alpha) multiplied by (up+1)) rather than GELU… show evidenceDense and expert MLPs compute a SiLU-sigmoid gated SwiGLU-OAI variant (gate * sigmoid(gate*alpha) multiplied by (up+1)) rather than GELU gating. text_config.hidden_act = "swigluoai" |
token mixer 17
softmax attention 2
grouped-query attention 2
“MSA is built upon Grouped Query Attention (GQA).”
“The full attention stack gets replaced with block-sparse GQA.”
sparse attention 15
“Context Scaling via Sparse Attention: M3 introduces MiniMax Sparse Attention (MSA) to improve long context efficiency.”
“adopting a "KV outer gather Q" approach that uses KV blocks as the outer loop to aggregate the queries that hit them”
“Most likely natively trained sparsity.”
“MSA follows the sparse softmax attention paradigm”
“MSA instantiates the two-stage sparse-attention formulation”
block selection 6
“We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA).”
“block-level selection, but attention computed on the real K/V.”
“dynamic sparse selection”
“the Main Branch then performs exact block-sparse attention over only the selected blocks.”
“selects the top- blocks via max-pooling scoring”
“Block Max Pool then compresses the token-level scores into block-level scores”
sparse attention indexer 3
“a lightweight Index Branch scores key–value blocks and independently selects a Top- subset for each GQA group”
“We use a two-stage training schedule to initialise the Index Branch and avoid early random selections.”
“K_idx has only one head — all heads share the same index key.”
fixed-pattern sparse attention 1
“the local block containing is always selected as part of during both training and inference.”
channel mixer 4
mixture of experts 4
“109B-parameter Mixture of Experts (MoE) model trained from scratch”
expert routing 1
“Each MoE layer uses 128 routed experts, 1 shared expert, and top-4 routed expert selection.”
expert load balancing 1
“employ persistent load balancing in the backward pass”
positional encoding 1
“RoPE dimension 64”
multimodal architecture 1
“M3 is a model that has undergone mixed-modality training from Step 0.”
7 more mentioned or evaluated
sparse attention “Compared with approaches like DSA and MoBA, MSA can partition the KV into blocks more precisely”
sliding window attention “a FLOP-matched sliding-window baseline”
global attention “MSA matched full attention on the vast majority of capabilities”
block selection “Compared with approaches like DSA and MoBA, MSA can partition the KV into blocks more precisely”
linear attention & state space “Hybrid architectures replace a subset of softmax attention layers with efficient alternatives such as linear attention”
hybrid layer stacking “the sliding window is most likely replaced by RoPE + attention sink, or simply by dense attention as a per-layer fallback”
attention sink “the sliding window is most likely replaced by RoPE + attention sink, or simply by dense attention as a per-layer fallback”
Not used 8
multi-head latent attention “GQA as the substrate, not MLA.”
attention sink “we do not include the learnable attention sink in the final recipe”
sparse attention “block-level selection like CSA but attention is computed on the real K/V.”
sparse attention “NSA originally has three parallel paths (compress + select + sliding window) plus a learned gate. M3 keeps only selection.”
sparse attention indexer “the Index Branch output is added to the layer output”
sparse attention indexer “drops the index value head on efficiency grounds”
fixed-pattern sparse attention “the final recipe does not force the first block or a large local window”
linear attention & state space “M1's Lightning Attention is unlikely to make a return”
auxiliary loss 3
“KL loss directly supervises the Index Branch by matching its selection distribution”
“stopping the KL gradient at the Index Branch input”
“LM Loss + KL Loss gives the best balance”
training runtime 1
“autotune configuration generation”
data mixture & curriculum 2
“M3 is a model that has undergone mixed-modality training from Step 0. This native multimodal approach allows the semantic spaces of…”
“So-called interleaved data, where text and images are woven together within a sequence, turned out to matter more than initially expected.”
agentic post-training 1
“MiniMax built a simulator framework that mimics typical behavior patterns”
reasoning control 3
“M3 evaluation: 512k max output tokens, temperature = 1.0, test-time-scaling framework up to 10 iterations.”
“M3 supports toggling thinking on or off. With thinking enabled, the model is suited to complex reasoning, agentic tasks, and long-horizon…”
“M3 automatically determines when additional reasoning is beneficial”
KV cache management 3
“The stored context, known as the key-value cache (KV cache), gets split into blocks.”
“Full automatic Cache support, no configuration needed”
“splits each KV tile along its query dimension into chunks”
inference scheduling 1
“host-side scheduling optimization”
inference kernel 6
“including baseline implementation, autotune configuration generation, performance bottleneck diagnosis, CUDA Graph integration, persistent…”
“MSA's kernel, described in section 4, reuses the FlashAttention algorithmic skeleton”
“The kernel runs as a persistent grid in which CTAs claim work through a global atomic counter”
“design an exp-free TopK kernel specialized for the small-k regime”
“persistent kernel rewriting”
“The forward is therefore split into two kernels”
agentic scaffolding 2
“Through a Producer + Verifier adversarial harness loop, the Agent Team can continuously produce, reflect, and correct itself during…”
“Apex-Agents: Uses archipelago codebase, ReAct Toolbelt framework, scoring model Claude Sonnet 4.6.”
1 more mentioned or evaluated
inference kernel “FP8 matrix multiplication (GEMM) is one of the most compute-intensive parts of large model inference, and also one of the most difficult…”
inference engine 4
“We recommend the following inference frameworks to serve the model: vLLM”
“We recommend the following inference frameworks to serve the model: SGLang”
“We recommend the following inference frameworks to serve the model: KTransformers”
“We recommend the following inference frameworks to serve the model: unsloth”
training framework 1
“We recommend the following inference frameworks to serve the model: Transformers”
agent product 4
“MiniMax Code is built on a harness based on the outstanding open-source community projects OpenCode and Pi.”
“SWE-Bench Verified: Tested on internal infrastructure using Claude Code as the scaffolding.”
“Claude Opus 4.7, MiniMax-M2.7, and MiniMax-M3 used Mini-SWE-Agent as scaffolding”
“MiniMax Code is built on a harness based on the outstanding open-source community projects OpenCode and Pi.”
2 more mentioned or evaluated
kernel & quantization library “more than 4× faster than the open-source Flash-Sparse-Attention and flash-moba”
kernel & quantization library “more than 4× faster than the open-source Flash-Sparse-Attention and flash-moba”
judge 2
“LLM-as-a-Judge scoring”
“Claude Code used as scaffolding, with Agent-as-a-Verifier paradigm for automated verification”
unfiled 7
“multimodal training from step zero achieving deep alignment between textual and visual semantic spaces”
“using Terminus 2 as the scaffolding”
“Apex-Agents: Uses archipelago codebase, ReAct Toolbelt framework”
“we optimize this by emitting these LSE values directly to global memory during the main pass”
“packs query positions together with their associated query heads”
“PostTrainBench: Evaluated on Claude Code using Ralph-Loop mechanism for 12 hours”
1 more mentioned or evaluated
“We have seen that Claude Code has also recently released Dynamic Workflows in a similar direction.”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- MiniMax-M3 architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| minimax.io/blog/minimax-m3 | official blog | MiniMax | read |
| arxiv.org/abs/2606.13392 | technical report | MiniMax | read |
| github.com/MiniMax-AI/MiniMax-M3 | code repo | MiniMax | read |
| huggingface.co/MiniMaxAI/MiniMax-M3 | model card | MiniMax | read |
| minimax.io/models/text/m3 | vendor docs | MiniMax | read |
| the-decoder.com/minimax-m3-open-weight-model-with-a-million-token-conte… | news | The Decoder | read |
| huggingface.co/blog/AtlasCloud-AI/minimax-goes-sparse | third party analysis | Hugging Face community blog | read |
| recipes.vllm.ai/MiniMaxAI/MiniMax-M3 | vendor docs | vLLM | not fetched |