Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionblock selection

specific method · filed under model architecture

MiniMax Sparse Attention

MiniMax Sparse Attention (MSA) is a GQA-based sparse attention mechanism featuring two branches.1 For each query token, a lightweight Index Branch selects a small set of key blocks, and the Main Branch computes softmax attention over those tokens.2

Trade-off MSA matches GQA on downstream benchmarks while delivering prefill and decoding speedups at 1M context length.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We introduce MiniMax Sparse Attention (MSA), a GQA-based sparse attention mechanism with two branches, as illustrated in Figure 1 — MiniMax Sparse Attention
  2. For each query token, a lightweight Index Branch selects a small set of key blocks from the causal context, and the Main Branch computes softmax attention over the tokens in those blocks — MiniMax Sparse Attention
  3. MSA matches GQA on downstream benchmarks while delivering prefill and decoding speedups at 1M context length — MiniMax Sparse Attention

Also called MSA (MiniMax Sparse Attention), MiniMax Sparse Attention (MSA), sequential block processing.

sources
7
models
2
lab adopt it
1
strongest
core

Documented in

Evidence

8 quotes from 7 sources, one card per model, each opening with its strongest.

How sources treat it
core 8

MiniMax-M3

MiniMaxadopted6 sources

We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA).

coremodel architecturein MiniMax-M3MiniMax
6 more from this model

MSA is a clean and easily extensible new sparse attention architecture. It gives M3 a 1M context window

coremodel architecturein MiniMax-M3MiniMax

M3 introduces MiniMax Sparse Attention (MSA) to improve long context efficiency

coremodel architecturein MiniMax-M3MiniMax

The technical foundation is a new attention variant called MiniMax Sparse Attention (MSA).

coremodel architecturein MiniMax-M3MiniMax

MSA flips the logic and processes blocks sequentially.

coresoftware implementationin MiniMax-M3MiniMax

Powered by the proprietary MiniMax Sparse Attention (MSA) architecture

coremodel architecturein MiniMax-M3MiniMax

M3 is powered by MiniMax Sparse Attention (MSA)

coremodel architecturein MiniMax-M3MiniMax

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑