implementation detail · filed under model architecture
Sparse expert activation
The implementation detail of activating only a subset of a model’s parameters or experts, without evidence for a distinct named routing method.
Also called Active parameters (sparse activation), active parameters, Sparse activation, sparse active-parameter MoE inference, Efficient sparse architecture, Mixture of Experts (MoE) sparsity.
- sources
- 8
- models
- 5
- labs adopt it
- 5
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Mixtral of Experts (Jiang et al., 2024) paper arxiv.orgconcrete open-weight top-2-of-8 routed activation
Evidence
8 spans quoted from the sources, strongest treatment first.
only 42 billion active parameters are used during inference.
By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might suggest.
activates just 16 of its 896 experts per token
The model activates roughly 6.8% of its parameters on any given token.
activates only 17 billion of its 397 billion total parameters per forward pass
Efficient sparse architecture: Laguna S 2.1 has 118B total parameters and activates 8B parameters per token.
By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might suggest.
We have also scaled up Mixture of Experts (MoE) sparsity, effectively activating 16 out of 896 experts when paired with a Stable LatentMoE framework.
Filed alongside
Other methods under model architecture :: channel mixer :: mixture of experts.