Open Models Almanac
ModelsMiniMax-M3

MiniMax

MiniMax-M3

What MiniMax-M3’s own documents say it is built from — every method with the sentence that describes it.

Parameters
~428B / ~23B active
Context
1M
Attention
sparse (indexer)MiniMax Sparse Attention (MSA) with GQA
Experts
128 routed + 1 shared, top-4
Input
text, image, video
Licence
MiniMax Community License
Released
2026-06-01

Against the consensus recipe

Its documents state 3 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 9 of the 19 architecture features checked — 6 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_minimax_m3_vl.py, configuration_minimax_m3_vl.py) and config.json at revision d95e30cb91, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention in its codeadopted

The language model uses 64 query heads sharing 4 key/value heads via repeat_kv grouping.

show evidencetext_config.num_attention_heads = 64text_config.num_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_minimax_m3_vl.py · L406
self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_minimax_m3_vl.py · L410–412
Multi-head latent attention not in its codenot used

Attention uses standard per-head q/k/v projections with GQA, with no latent low-rank KV projection or lora rank config present.

Sliding-window attention not in its codediscussed

No sliding_window config key or windowed masking code is present; layer_types only distinguishes full_attention from the sparse indexer…

show the whole verdict

No sliding_window config key or windowed masking code is present; layer_types only distinguishes full_attention from the sparse indexer layers.

Interleaved sliding-window and global attention not in its codenot stated

layer_types alternates only between full_attention and the lightning-indexer sparse attention, not sliding-window vs global attention.

Indexer-selected sparse attention (DeepSeek Sparse Attention) code only in its codediscussed

A lightning indexer scores past tokens in blocks and selects top-k blocks per query, enabled on most layers via…

show evidence

A lightning indexer scores past tokens in blocks and selects top-k blocks per query, enabled on most layers via sparse_attention_freq/layer_types.

text_config.sparse_attention_config.use_sparse_attention = truetext_config.sparse_attention_config.sparse_attention_freq = [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
topk = min(self.topk_blocks, num_key_blocks)
        topk_scores, topk_indices = block_scores.topk(topk, dim=-1)  # [B, H_idx, S_q, topk]
        return topk_indices.masked_fill(topk_scores == float("-inf"), -1)
modeling_minimax_m3_vl.py · L583–585
self.indexer = (
            MiniMaxM3VLIndexer(config, layer_idx) if config.layer_types[layer_idx] == "minimax_m3_sparse" else None
        )
modeling_minimax_m3_vl.py · L416–418
Linear-attention or state-space layers alongside full attention not in its codenot stated

There is no recurrent/state-space linear-attention layer type; only softmax attention (full or sparse-indexer-selected) layers exist.

Gated DeltaNet layers not in its codenot stated

No gated delta rule recurrent mixer or related config keys (linear_num_key_heads, linear_conv_kernel_dim) appear in this model.

Mamba-2 layers not in its codenot stated

No Mamba-2 SSM blocks or related config keys (mamba_num_heads, ssm_state_size) are present.

Learnable attention sink not in its codenot stated

The attention softmax has no extra learned sink logit; eager_attention_forward only uses the standard query/key/value matmuls.

Gated attention output not in its codenot stated

attention_output_gate is set false in text_config and no sigmoid gating of attention output is implemented before o_proj.

show evidencetext_config.attention_output_gate = false
QK normalization code only in its codenot stated

use_qk_norm is true and the attention module applies per-head RMSNorm to queries and keys before RoPE/dot-product.

show evidencetext_config.use_qk_norm = truetext_config.qk_norm_type = "per_head"
query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_minimax_m3_vl.py · L431–432
Partial RoPE code only in its codenot stated

partial_rotary_factor is 0.5 and apply_rotary_pos_emb only rotates the first rotary_dim channels, passing the rest through unrotated.

show evidencetext_config.partial_rotary_factor = 0.5text_config.rotary_dim = 64
rotary_dim = cos.shape[-1]
    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
modeling_minimax_m3_vl.py · L375–377
YaRN RoPE scaling code unclearnot stated

rope_parameters is not given a literal value in the provided config.json (it is None in the class default) so whether YaRN scaling is…

show evidence

rope_parameters is not given a literal value in the provided config.json (it is None in the class default) so whether YaRN scaling is active cannot be determined from these files.

self.rope_type = self.config.rope_parameters["rope_type"]
        rope_init_fn: Callable = self.compute_default_rope_parameters
        if self.rope_type != "default":
            rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
modeling_minimax_m3_vl.py · L273–276
Layers without positional encoding (NoPE) not in its codenot stated

All attention layers apply the same rotary embedding via apply_rotary_pos_emb; there is no per-layer toggle to skip positional encoding…

show the whole verdict

All attention layers apply the same rotary embedding via apply_rotary_pos_emb; there is no per-layer toggle to skip positional encoding entirely.

Mixture of experts in its codeadopted

MoE feed-forward layers route each token to 4 of 128 experts via a learned sigmoid-scored router, enabled on most layers via…

show evidence

MoE feed-forward layers route each token to 4 of 128 experts via a learned sigmoid-scored router, enabled on most layers via mlp_layer_types.

text_config.num_local_experts = 128text_config.num_experts_per_tok = 4
_, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_minimax_m3_vl.py · L236–237
Shared expert in its codeadopted

Each MoE block runs a dense shared expert of shared_intermediate_size=3072 (n_shared_experts=1) whose output is added to the routed…

show evidence

Each MoE block runs a dense shared expert of shared_intermediate_size=3072 (n_shared_experts=1) whose output is added to the routed experts' output.

text_config.n_shared_experts = 1text_config.shared_intermediate_size = 3072
self.shared_experts = MiniMaxM3VLDenseMLP(config, intermediate_size=config.shared_intermediate_size)
modeling_minimax_m3_vl.py · L248
hidden_states = hidden_states + shared_output
modeling_minimax_m3_vl.py · L259
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds e_score_correction_bias to the sigmoid scores solely to pick top-k experts, while actual routing weights are taken from…

show evidence

The router adds e_score_correction_bias to the sigmoid scores solely to pick top-k experts, while actual routing weights are taken from the unbiased scores.

text_config.use_routing_bias = true
scores_for_choice = routing_weights + self.e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_minimax_m3_vl.py · L235–237
Multi-token prediction layers code only in its codenot stated

The config declares num_mtp_modules=7 and num_nextn_predict_layers=1, and the model explicitly ignores mtp.* weights on load, confirming…

show evidence

The config declares num_mtp_modules=7 and num_nextn_predict_layers=1, and the model explicitly ignores mtp.* weights on load, confirming MTP modules are part of the checkpoint.

text_config.num_mtp_modules = 7text_config.num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_minimax_m3_vl.py · L686
SwiGLU feed-forward code only in its codenot stated

Dense and expert MLPs compute a SiLU-sigmoid gated SwiGLU-OAI variant (gate * sigmoid(gate*alpha) multiplied by (up+1)) rather than GELU…

show evidence

Dense and expert MLPs compute a SiLU-sigmoid gated SwiGLU-OAI variant (gate * sigmoid(gate*alpha) multiplied by (up+1)) rather than GELU gating.

text_config.hidden_act = "swigluoai"
glu = gate * torch.sigmoid(gate * self.swiglu_alpha)
        return self.down_proj((up + 1.0) * glu)
modeling_minimax_m3_vl.py · L176–177

token mixer 17

softmax attention 2

grouped-query attention 2

Grouped-query attention core

“MSA is built upon Grouped Query Attention (GQA).”

Block-sparse grouped-query attention core

“The full attention stack gets replaced with block-sparse GQA.”

sparse attention 15

Sparse attention core

“Context Scaling via Sparse Attention: M3 introduces MiniMax Sparse Attention (MSA) to improve long context efficiency.”

KV-outer sparse attention used

“adopting a "KV outer gather Q" approach that uses KV blocks as the outer loop to aggregate the queries that hit them”

Natively trained sparsity used

“Most likely natively trained sparsity.”

Sparse softmax attention used

“MSA follows the sparse softmax attention paradigm”

Two-stage sparse-attention formulation used

“MSA instantiates the two-stage sparse-attention formulation”

block selection 6

MiniMax Sparse Attention core

“We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA).”

Block-level selection core

“block-level selection, but attention computed on the real K/V.”

Dynamic sparse selection core

“dynamic sparse selection”

Main Branch core

“the Main Branch then performs exact block-sparse attention over only the selected blocks.”

Top-k block selection used

“selects the top- blocks via max-pooling scoring”

Block Max Pooling used

“Block Max Pool then compresses the token-level scores into block-level scores”

sparse attention indexer 3

Index Branch core

“a lightweight Index Branch scores key–value blocks and independently selects a Top- subset for each GQA group”

Indexer warmup used

“We use a two-stage training schedule to initialise the Index Branch and avoid early random selections.”

Single-head index key used

“K_idx has only one head — all heads share the same index key.”

fixed-pattern sparse attention 1

Local Block used

“the local block containing is always selected as part of during both training and inference.”

channel mixer 4

mixture of experts 4

Mixture of experts core

“109B-parameter Mixture of Experts (MoE) model trained from scratch”

expert routing 1

Top-k expert routing used

“Each MoE layer uses 128 routed experts, 1 shared expert, and top-4 routed expert selection.”

expert load balancing 1

Persistent load balancing used

“employ persistent load balancing in the backward pass”

shared experts 1

Shared expert used

“Each MoE layer uses 128 routed experts, 1 shared expert, and top-4 routed expert selection.”

positional encoding 1

Rotary Position Embedding used

“RoPE dimension 64”

multimodal architecture 1

Mixed-modality training from the start core

“M3 is a model that has undergone mixed-modality training from Step 0.”

7 more mentioned or evaluated
DeepSeek Sparse Attention evaluated

sparse attention “Compared with approaches like DSA and MoBA, MSA can partition the KV into blocks more precisely”

Sliding Window Attention evaluated

sliding window attention “a FLOP-matched sliding-window baseline”

Full attention evaluated

global attention “MSA matched full attention on the vast majority of capabilities”

Mixture of Block Attention evaluated

block selection “Compared with approaches like DSA and MoBA, MSA can partition the KV into blocks more precisely”

Linear attention mentioned

linear attention & state space “Hybrid architectures replace a subset of softmax attention layers with efficient alternatives such as linear attention”

Dense Attention Fallback unclear

hybrid layer stacking “the sliding window is most likely replaced by RoPE + attention sink, or simply by dense attention as a per-layer fallback”

RoPE with attention sink unclear

attention sink “the sliding window is most likely replaced by RoPE + attention sink, or simply by dense attention as a per-layer fallback”

Not used 8

Multi-head Latent Attention not used

multi-head latent attention “GQA as the substrate, not MLA.”

Learnable attention sink token not used

attention sink “we do not include the learnable attention sink in the final recipe”

Compressed Sparse Attention not used

sparse attention “block-level selection like CSA but attention is computed on the real K/V.”

Native Sparse Attention not used

sparse attention “NSA originally has three parallel paths (compress + select + sliding window) plus a learned gate. M3 keeps only selection.”

Index Branch output not used

sparse attention indexer “the Index Branch output is added to the layer output”

Index value head not used

sparse attention indexer “drops the index value head on efficiency grounds”

Forced Sink and Local Window not used

fixed-pattern sparse attention “the final recipe does not force the first block or a large local window”

Lightning Attention not used

linear attention & state space “M1's Lightning Attention is unlikely to make a return”

auxiliary loss 3

KL alignment loss used

“KL loss directly supervises the Index Branch by matching its selection distribution”

Gradient detach used

“stopping the KL gradient at the Index Branch input”

Language modeling loss plus KL loss used

“LM Loss + KL Loss gives the best balance”

training runtime 1

Autotune configuration generation used

“autotune configuration generation”

data mixture & curriculum 2

Interleaved mixed-modality training used

“M3 is a model that has undergone mixed-modality training from Step 0. This native multimodal approach allows the semantic spaces of…”

Interleaved multimodal data used

“So-called interleaved data, where text and images are woven together within a sequence, turned out to matter more than initially expected.”

agentic post-training 1

Multi-turn collaboration simulator used

“MiniMax built a simulator framework that mimics typical behavior patterns”

reasoning control 3

Test-time compute scaling used

“M3 evaluation: 512k max output tokens, temperature = 1.0, test-time-scaling framework up to 10 iterations.”

Thinking mode optional

“M3 supports toggling thinking on or off. With thinking enabled, the model is suited to complex reasoning, agentic tasks, and long-horizon…”

Adaptive reasoning optional

“M3 automatically determines when additional reasoning is beneficial”

KV cache management 3

Block-based KV cache core

“The stored context, known as the key-value cache (KV cache), gets split into blocks.”

Automatic cache support default

“Full automatic Cache support, no configuration needed”

Pre-scheduled tile chunking used

“splits each KV tile along its query dimension into chunks”

inference scheduling 1

Host-side scheduling optimization used

“host-side scheduling optimization”

inference kernel 6

CUDA Graph capture and replay used

“including baseline implementation, autotune configuration generation, performance bottleneck diagnosis, CUDA Graph integration, persistent…”

FlashAttention used

“MSA's kernel, described in section 4, reuses the FlashAttention algorithmic skeleton”

Dynamic load balancing used

“The kernel runs as a persistent grid in which CTAs claim work through a global atomic counter”

Exp-free TopK kernel used

“design an exp-free TopK kernel specialized for the small-k regime”

Persistent kernel used

“persistent kernel rewriting”

Two-phase forward used

“The forward is therefore split into two kernels”

agentic scaffolding 2

Producer–Verifier adversarial harness loop used

“Through a Producer + Verifier adversarial harness loop, the Agent Team can continuously produce, reflect, and correct itself during…”

ReAct Toolbelt used

“Apex-Agents: Uses archipelago codebase, ReAct Toolbelt framework, scoring model Claude Sonnet 4.6.”

1 more mentioned or evaluated
FP8 GEMM evaluated

inference kernel “FP8 matrix multiplication (GEMM) is one of the most compute-intensive parts of large model inference, and also one of the most difficult…”

inference engine 4

vLLM optional

“We recommend the following inference frameworks to serve the model: vLLM”

SGLang optional

“We recommend the following inference frameworks to serve the model: SGLang”

KTransformers optional

“We recommend the following inference frameworks to serve the model: KTransformers”

Unsloth optional

“We recommend the following inference frameworks to serve the model: unsloth”

training framework 1

Hugging Face Transformers optional

“We recommend the following inference frameworks to serve the model: Transformers”

agent product 4

OpenCode used

“MiniMax Code is built on a harness based on the outstanding open-source community projects OpenCode and Pi.”

Claude Code used

“SWE-Bench Verified: Tested on internal infrastructure using Claude Code as the scaffolding.”

Mini-SWE-Agent used

“Claude Opus 4.7, MiniMax-M2.7, and MiniMax-M3 used Mini-SWE-Agent as scaffolding”

Pi used

“MiniMax Code is built on a harness based on the outstanding open-source community projects OpenCode and Pi.”

2 more mentioned or evaluated
flash-moba evaluated

kernel & quantization library “more than 4× faster than the open-source Flash-Sparse-Attention and flash-moba”

Flash-Sparse-Attention evaluated

kernel & quantization library “more than 4× faster than the open-source Flash-Sparse-Attention and flash-moba”

judge 2

LLM-as-a-Judge used

“LLM-as-a-Judge scoring”

Agent-as-a-Verifier used

“Claude Code used as scaffolding, with Agent-as-a-Verifier paradigm for automated verification”

unfiled 7

multimodal training from step zero core

“multimodal training from step zero achieving deep alignment between textual and visual semantic spaces”

Terminus 2 used

“using Terminus 2 as the scaffolding”

archipelago used

“Apex-Agents: Uses archipelago codebase, ReAct Toolbelt framework”

LSE fusion used

“we optimize this by emitting these LSE values directly to global memory during the main pass”

query concatenation used

“packs query positions together with their associated query heads”

Ralph-Loop mechanism used

“PostTrainBench: Evaluated on Claude Code using Ralph-Loop mechanism for 12 hours”

1 more mentioned or evaluated
Dynamic Workflows mentioned

“We have seen that Claude Code has also recently released Dynamic Workflows in a similar direction.”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
minimax.io/blog/minimax-m3official blogMiniMaxread
arxiv.org/abs/2606.13392technical reportMiniMaxread
github.com/MiniMax-AI/MiniMax-M3code repoMiniMaxread
huggingface.co/MiniMaxAI/MiniMax-M3model cardMiniMaxread
minimax.io/models/text/m3vendor docsMiniMaxread
the-decoder.com/minimax-m3-open-weight-model-with-a-million-token-conte…newsThe Decoderread
huggingface.co/blog/AtlasCloud-AI/minimax-goes-sparsethird party analysisHugging Face community blogread
recipes.vllm.ai/MiniMaxAI/MiniMax-M3vendor docsvLLMnot fetched
↑