Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsfine-grained experts

single mention · model architecture

Fine-grained routed experts with a shared expert

A mixture-of-experts design that replaces the FFN with a lightweight shared expert plus a set of fine-grained routed experts, each with a smaller intermediate width.1,2 The shared expert captures general principles, while each token independently activates multiple routed experts via sigmoid-based routing for specialized modeling.3,4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we adopt fine-grained expert segmentation and shared expert isolation, in which a lightweight shared expert preserves universal priors, while a set of routed experts provides specialized modeling capacity — From Foundation to Application: Improving VLA Models in Practice
  2. both the shared and routed experts employ a smaller intermediate width — From Foundation to Application: Improving VLA Models in Practice
  3. employing sigmoid-based routing confidence to allow each token to independently activate multiple experts — From Foundation to Application: Improving VLA Models in Practice
  4. encouraging shared experts to capture general principles and routed experts to provide stronger specialization — From Foundation to Application: Improving VLA Models in Practice

Also called shared and fine-grained routed experts.

Filed under model architecture › channel mixer › mixture of experts › fine-grained experts