ModelsMiniMax-M3

MiniMax · usage rank #12 · released 2026-06-01

MiniMax-M3

What MiniMax-M3’s own documents say it is built from — every method with the sentence that describes it.

Curator’s note~428B total / ~23B active, 1M context, native multimodal, MiniMax Community License (open-weight, not OSI open-source). arXiv 2606.13392 is the MiniMax Sparse Attention (MSA) paper — the architecture behind M3 rather than a full M3 system report.

Rank by open-weight tokens on OpenRouter, week of 2026-09-21.

Against the consensus recipe

Its documents state 3 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

Data curation
- Agentic data synthesis pipelinenot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- KL alignment lossused
Model architecture
- Mixture of Expertscore
- Hybrid Attentionnot in its documents
- Multi-Token Predictionnot in its documents
Optimization
- Muonnot in its documents
- Expert Parallelismnot in its documents
- Cosine decaynot in its documents
Post-training
- Group Relative Policy Optimizationnot in its documents
- Supervised fine-tuningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Configurable reasoning effortoptional
- Speculative decodingnot in its documents
- FP8 KV-cache quantizationnot in its documents

Checked in its code

Its code shows 9 of the 19 architecture features checked — 6 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_minimax_m3_vl.py, configuration_minimax_m3_vl.py) and config.json at revision f0e1c1e04d, by anthropic/claude-sonnet-5 on 2026-09-25.

Feature	Code	Its documents	Evidence
Grouped-query attention	in its code	core	The text attention uses 64 query heads with only 4 key/value heads, repeated via repeat_kv/num_key_value_groups. text_config.num_attention_heads = 64 text_config.num_key_value_heads = 4 `self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads` modeling_minimax_m3_vl.py · L423
Multi-head latent attention	not in its code	not used	There is no latent KV down-projection/up-projection (no kv_lora_rank/q_lora_rank in config); attention uses standard per-head K/V projections instead.
Sliding-window attention	not in its code	mentioned	No sliding_window config key or windowed-mask code exists; layer_types only distinguish full_attention vs the sparse indexer layers, not a local window.
Interleaved sliding-window and global attention	not in its code	not stated	layer_types alternates between full_attention and the sparse-indexer attention, not between sliding-window and global attention, so this interleaving pattern is absent. `self.layer_types = [ "minimax_m3_sparse" if f else "full_attention" for f in sparse_cfg["sparse_attention_freq"] ]` configuration_minimax_m3_vl.py · L144–146
Indexer-selected sparse attention (DeepSeek Sparse Attention) code only	in its code	evaluated	A lightning indexer scores keys per head and pools into blocks, selecting the top index_topk_blocks blocks per query for sparse attention on sparse_attention_freq==1 layers. text_config.sparse_attention_config.use_sparse_attention = true text_config.sparse_attention_config.sparse_topk_blocks = 16 `topk = min(self.topk_blocks, num_key_blocks) topk_scores, topk_indices = block_scores.topk(topk, dim=-1) # [B, H_idx, S_q, topk] return topk_indices.masked_fill(topk_scores == float("-inf"), -1)` modeling_minimax_m3_vl.py · L600–602
Linear-attention or state-space layers alongside full attention	not in its code	not stated	No recurrent linear-attention or SSM layer type appears in layer_types or the code; all token-mixing layers are softmax attention (full or sparse-indexed).
Gated DeltaNet layers	not in its code	not stated	No gated delta-rule recurrent mixer is implemented; only standard attention and the lightning indexer exist.
Mamba-2 layers	not in its code	not stated	No Mamba-2/SSD state-space block implementation or config keys are present.
Learnable attention sink	not in its code	not used	The eager attention forward computes a plain softmax over query-key scores with no extra learned sink logit column. `attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)` modeling_minimax_m3_vl.py · L362
Gated attention output	not in its code	not stated	attention_output_gate is set false in the config and no sigmoid gating of the attention output before o_proj appears in the attention forward. text_config.attention_output_gate = false
QK normalization code only	in its code	not stated	Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before the rotary embedding and attention dot product, and use_qk_norm is true. text_config.use_qk_norm = true text_config.qk_norm_type = "per_head" `query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2) key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)` modeling_minimax_m3_vl.py · L448–449
Partial RoPE code only	in its code	not stated	partial_rotary_factor is 0.5 and apply_rotary_pos_emb only rotates the first rotary_dim channels of each head, leaving the remainder untouched. text_config.partial_rotary_factor = 0.5 text_config.rotary_dim = 64 `rotary_dim = cos.shape[-1] q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:] k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]` modeling_minimax_m3_vl.py · L392–394
YaRN RoPE scaling	code unclear	not stated	rope_parameters is not set in the given config (it is null/None by default) so whether YaRN scaling type is configured cannot be determined from these files.
Layers without positional encoding (NoPE)	not in its code	not stated	Every attention layer (whether full_attention or minimax_m3_sparse) receives the same rotary position_embeddings applied via apply_rotary_pos_emb; there is no per-layer toggle to skip RoPE. `cos, sin = position_embeddings query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)` modeling_minimax_m3_vl.py · L452–453
Mixture of experts	in its code	core	Feed-forward layers marked 'sparse' in mlp_layer_types route each token to num_experts_per_tok of num_local_experts via a sigmoid top-k router. text_config.num_local_experts = 128 text_config.num_experts_per_tok = 4 `_, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False) top_k_weights = routing_weights.gather(1, top_k_index)` modeling_minimax_m3_vl.py · L243–244
Shared expert	in its code	used	Each MoE block also runs a dense shared expert (shared_intermediate_size, n_shared_experts=1) whose output is added to the routed-expert output. text_config.n_shared_experts = 1 text_config.shared_intermediate_size = 3072 `self.shared_experts = MiniMaxM3VLDenseMLP(config, intermediate_size=config.shared_intermediate_size)` modeling_minimax_m3_vl.py · L255
Auxiliary-loss-free load balancing (selection bias) code only	in its code	not stated	The router adds an e_score_correction_bias only to the scores used for top-k selection, not to the routing weights used for output scaling, matching DeepSeek-V3 style bias. text_config.use_routing_bias = true `scores_for_choice = routing_weights + self.e_score_correction_bias _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False) top_k_weights = routing_weights.gather(1, top_k_index)` modeling_minimax_m3_vl.py · L242–244
Multi-token prediction layers code only	in its code	not stated	The config declares num_mtp_modules=7 and num_nextn_predict_layers=1, and the model explicitly ignores mtp.* weights on load, indicating declared but skipped MTP modules. text_config.num_mtp_modules = 7 text_config.num_nextn_predict_layers = 1 `_keys_to_ignore_on_load_unexpected = [r"(^\|\.)mtp\..*"]` modeling_minimax_m3_vl.py · L703
SwiGLU feed-forward code only	in its code	not stated	Both the dense MLP and expert MLPs compute a SiLU-sigmoid-gated GLU (gate * sigmoid(gatealpha)) multiplied by up, i.e. a SwiGLU variant (swigluoai), not GeGLU. text_config.hidden_act = "swigluoai" `glu = gate torch.sigmoid(gate * self.swiglu_alpha) return self.down_proj((up + 1.0) * glu)` modeling_minimax_m3_vl.py · L183–184

model architecture 39

token mixer 33

softmax attention 8

channel mixer 4

mixture of experts 4

Mixture of Experts core

expert routing 1

Top-4 expert routing used

expert load balancing 1

Persistent load balancing used

shared experts 1

Shared Experts used

Language modeling loss plus KL loss used

Thinking mode selection optional

Adaptive reasoning optional

KV cache management 3

Block-based KV cache core

Automatic cache default

Pre-scheduled tile chunking used

inference scheduling 1

Host-side scheduling optimization used

inference kernel 7

CUDA Graph used

Dynamic load balancing used

Exp-free TopK kernel used

FlashAttention used

Persistent kernel used

Two-phase forward used

FP8 GEMM evaluated

KTransformers optional

Unsloth optional

Mini-SWE-Agent used

Pi used

evaluation 2

judge 2

Agent-as-a-Verifier used

LLM-as-a-Judge used

unfiled 7

multimodal training from step zero core

Terminus 2 used

archipelago used

LSE fusion used

query concatenation used

Ralph-Loop mechanism used

Dynamic Workflows mentioned

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

Document	Kind	Publisher	Status
minimax.io/blog/minimax-m3	official blog	MiniMax	read
arxiv.org/abs/2606.13392	technical report	MiniMax	read
github.com/MiniMax-AI/MiniMax-M3	code repo	MiniMax	read
huggingface.co/MiniMaxAI/MiniMax-M3	model card	MiniMax	read
minimax.io/models/text/m3	vendor docs	MiniMax	read
the-decoder.com/minimax-m3-open-weight-model-with-a-million-token-conte…	news	The Decoder	read
huggingface.co/blog/AtlasCloud-AI/minimax-goes-sparse	third party analysis	Hugging Face community blog	read

MiniMax-M3

Its documents state 3 of the 15 methods the field agrees on.

Data curation

Training objective

Model architecture

Optimization

Post-training

Inference & serving

Its code shows 9 of the 19 architecture features checked — 6 of them are not stated in its documents.

model architecture 39

token mixer 33

softmax attention 8

global attention 1

sliding window attention 2

grouped-query attention 2

multi-head latent attention 1

attention sink 2

sparse attention 22

block selection 7

sparse attention indexer 5

fixed-pattern sparse attention 1

linear attention & state space 2

hybrid layer stacking 1

channel mixer 4

mixture of experts 4

expert routing 1

expert load balancing 1

shared experts 1

positional encoding 1

multimodal architecture 1

training objective 3

auxiliary loss 3

optimization 1

training runtime 1

data curation 2

data mixture & curriculum 2

post-training 1

agentic post-training 1

inference & serving 17

reasoning control 4

KV cache management 3

inference scheduling 1

inference kernel 7

agentic scaffolding 2

software implementation 11

inference engine 4

training framework 1

kernel & quantization library 2

agent product 4

evaluation 2

judge 2

unfiled 7

Further reading

Sources