Open Models Almanac
Methodsmodel architecturechannel mixerdense feed-forward network

single mention · model architecture

Leading dense feed-forward layers before MoE layers

An architecture placement in which the first two blocks are dense FFNs before MoE blocks.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See dense feed-forward network for the methods filed beside it.

Also called dense FFNs in its first two blocks.

Filed under model architecture › channel mixer › dense feed-forward network