Open Models Almanac
Methodsmodel architecturechannel mixermixture of experts

implementation detail · filed under model architecture

Sparse expert activation

The implementation detail of activating only a subset of model parameters or experts during inference or per token, without evidence for a distinct routing algorithm.

The sources on this page name it without explaining how it works. The line above is the extraction's one-line gloss. See mixture of experts for the methods filed beside it.

Also called Sparse activation, Efficient sparse architecture, Token-level sparse expert routing, routed experts, 192 routed experts (top-8), token-level expert routing.

sources
9
models
7
labs adopt it
7
strongest
core

Documented in

Alibaba (Qwen)
Google DeepMind
Moonshot AI
Poolside
Tencent (Hunyuan)

Evidence

9 quotes from 9 sources, one card per model, each opening with its strongest.

How sources treat it
core 9

Laguna-S-2.1

Poolsideadopted2 sources

Efficient sparse architecture: Laguna S 2.1 has 118B total parameters and activates 8B parameters per token.

coremodel architecturein Laguna S 2.1Poolside
1 more from this model

The model activates roughly 6.8% of its parameters on any given token.

coremodel architecturein Laguna S 2.1Poolside

Gemma 4 31B

Google DeepMindadopted2 sources

By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might suggest.

coremodel architecturein Gemma 4Google DeepMind
1 more from this model

By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might suggest.

coremodel architecturein Gemma 4Google DeepMind

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑