Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsfine-grained experts

general family · filed under model architecture

Expert granularity

The general design choice of using more, narrower experts or fewer, wider experts in a mixture-of-experts layer.

The sources on this page name it without explaining how it works. The line above is the extraction's one-line gloss. See fine-grained experts for the methods filed beside it.

Also called Granular Mixture of Experts, Granular MoEs, Mixture-of-experts expert granularity.

sources
3
models
2
labs adopt it
0
strongest
evaluated

Documented in

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1not used 2

Kolibri-1

Aleph Alphadiscussed1 source

We compare three expert configurations of about 80B total parameters

evaluatedmodel architecturein KolibriAleph Alpha

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAnot used2 sources

LatentMoE helped us achieve better accuracy per parameter than standard Granular MoEs

not usedmodel architecturein Nemotron 3 UltraNVIDIA
1 more from this model

LatentMoE helped us achieve better accuracy per parameter than standard Granular MoEs

not usedmodel architecturein Nemotron 3 UltraNVIDIA
↑