Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

general family · filed under model architecture

Sparse attention

Sparse attention factors causal attention into an indexer that selects which keys to attend to, followed by attention computed only over the selected keys.1 Selections are made per query position (or per GQA-group query head) rather than for the whole layer, so each query computes standard softmax attention restricted to its own index set.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. A sparse attention layer factors causal attention into an indexer that selects which keys to attend to and a sparse attention computation over the selected keys — MiniMax Sparse Attention
  2. In multi-head attention, each query, specified by a position and a query head , can select a different key/value index set — MiniMax Sparse Attention

Also called sparse attention layers, sparse attention throughout.

sources
8
models
4
labs adopt it
3
strongest
core

Documented in

Evidence

8 quotes from 8 sources, one card per model, each opening with its strongest.

How sources treat it
mentioned 1used 1core 6

MiniMax-M3

MiniMaxadopted5 sources

Context Scaling via Sparse Attention: M3 introduces MiniMax Sparse Attention (MSA) to improve long context efficiency.

coremodel architecturein MiniMax-M3MiniMax
4 more from this model

MSA is a clean and easily extensible new sparse attention architecture

coremodel architecturein MiniMax-M3MiniMax

uses sparse attention from the first step

coremodel architecturein MiniMax Sparse AttentionMiniMax

Context Scaling via Sparse Attention

mentionedmodel architecturein MiniMax-M3MiniMax

MiniMax Goes Sparse

coremodel architecturein MiniMax-M3MiniMax

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑