single mention · model architecture
Leading dense feed-forward layers before MoE layers
An architecture placement in which the first two blocks are dense FFNs before MoE blocks.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See dense feed-forward network for the methods filed beside it.
Also called dense FFNs in its first two blocks.
Filed under model architecture › channel mixer › dense feed-forward network