Open Models Almanac
Methodsmodel architecturetoken mixerhybrid layer stacking

single mention · model architecture

Hybrid sparse mixture-of-experts Transformer architecture

The sources on this page only mention it in passing. See hybrid layer stacking for the methods filed beside it.

Also called hybrid sparse MoE Transformer backbone.

Filed under model architecture › token mixer › hybrid layer stacking