single mention · model architecture
Many small experts
A configuration favoring 384 smaller experts over fewer wider experts, based on reported test performance.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See fine-grained experts for the methods filed beside it.
Filed under model architecture › channel mixer › mixture of experts › fine-grained experts