Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsfine-grained experts

single mention · model architecture

Granular Mixture-of-Experts with routed and shared experts

A granular MoE configuration with 128 routed experts, four active per token, and one shared expert.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See fine-grained experts for the methods filed beside it.

Filed under model architecture › channel mixer › mixture of experts › fine-grained experts