Kolibri-1
Aleph AlphaWe compare three expert configurations of about 80B total parameters
general family · filed under model architecture
The general design choice of using more, narrower experts or fewer, wider experts in a mixture-of-experts layer.
The sources on this page name it without explaining how it works. The line above is the extraction's one-line gloss. See fine-grained experts for the methods filed beside it.
Also called Granular Mixture of Experts, Granular MoEs, Mixture-of-experts expert granularity.
3 quotes from 3 sources, one card per model, each opening with its strongest.
We compare three expert configurations of about 80B total parameters
LatentMoE helped us achieve better accuracy per parameter than standard Granular MoEs
LatentMoE helped us achieve better accuracy per parameter than standard Granular MoEs