Model techniques map
Techniquesmodel architecturetoken mixersparse attention

general family · filed under model architecture

Sparse attention

A broad attention family in which attention is restricted to a sparse subset rather than computed densely.

Also called sparse attention layers.

sources
6
models
2
labs adopt it
2
strongest
core

How sources treat it

One count per evidence span, weakest treatment to strongest.

mentioned 1used 1core 4

Documented in

GLM-5.2MiniMax-M3

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Evidence

6 spans quoted from the sources, strongest treatment first.

MSA is a clean and easily extensible new sparse attention architecture

coremodel architecturein MiniMax-M3MiniMax
MiniMax M3: Frontier Coding, 1M Context, Native Multimodality — All in One Model - MiniMax Research·MiniMax·2026-05-31

uses sparse attention from the first step

coremodel architecturein MiniMax Sparse AttentionMiniMax
MiniMax Sparse Attention·MiniMax·2026-06-11

MiniMax Goes Sparse

coremodel architecturein MiniMax-M3MiniMax
MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram·Hugging Face community blog·2026-05-29

Context Scaling via Sparse Attention: M3 introduces MiniMax Sparse Attention (MSA) to improve long context efficiency.

coremodel architecturein MiniMax-M3MiniMax
MiniMaxAI/MiniMax-M3 · Hugging Face·MiniMax·2026-06-11

every four sparse attention layers

usedmodel architecturein GLM-5.2Z.ai
zai-org/GLM-5.2 · Hugging Face·Z.ai·2026-02-17

Context Scaling via Sparse Attention

mentionedmodel architecturein MiniMax-M3MiniMax
README·MiniMax

Filed alongside

Other methods under model architecture :: token mixer :: sparse attention.

DeepSeek Sparse AttentionCompressed Sparse AttentionQwen Sparse AttentionHeavily Compressed AttentionGated DeepSeek Sparse AttentionCSA2 Full ModeKV-outer sparse attentionProgressive sequence-length extension for sparse attentionSparse-attention continued pre-training with joint model and indexer optimizationCross-layer KV and index reuse with statically assigned CSA2 modesFixed-budget sparse-attention selectionFrom-scratch sparse attention training without dense warmupJoint backbone and indexer training under sparse attentionNative Sparse AttentionNatively trained sparsityNoPE sparse multi-head latent attentionQSA micro-block compression at ratio 4Sequential block processingSparse retrieval over long contextsSparse softmax attentionToken-wise compressionTwo-stage introduction of sparse attentionTwo-stage sparse attention