Laguna-S-2.1
PoolsideEfficient sparse architecture: Laguna S 2.1 has 118B total parameters and activates 8B parameters per token.
1 more from this model
The model activates roughly 6.8% of its parameters on any given token.
implementation detail · filed under model architecture
The implementation detail of activating only a subset of model parameters or experts during inference or per token, without evidence for a distinct routing algorithm.
The sources on this page name it without explaining how it works. The line above is the extraction's one-line gloss. See mixture of experts for the methods filed beside it.
Also called Sparse activation, Efficient sparse architecture, Token-level sparse expert routing, routed experts, 192 routed experts (top-8), token-level expert routing.
9 quotes from 9 sources, one card per model, each opening with its strongest.
Efficient sparse architecture: Laguna S 2.1 has 118B total parameters and activates 8B parameters per token.
The model activates roughly 6.8% of its parameters on any given token.
By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might suggest.
By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might suggest.
only 42 billion active parameters are used during inference.
MoEs are fast and efficient because a router sends each token to just a few of its many experts, so only a fraction of the model’s parameters run per token.
activates just 16 of its 896 experts per token
192 routed experts (top-8) + 1 shared expert
activates only 17 billion of its 397 billion total parameters per forward pass
Picked by hand, not extracted: where to read more, not evidence for anything on this page.