single mention · model architecture
Dense-first-layer, sparse-MoE-later-layer placement
A layer-placement scheme in which initial transformer layers are dense and later layers contain MoE blocks.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See mixture of experts for the methods filed beside it.
Filed under model architecture › channel mixer › mixture of experts