single mention · model architecture
Fine-grained routed experts with a shared expert
A mixture-of-experts design that replaces the FFN with a lightweight shared expert plus a set of fine-grained routed experts, each with a smaller intermediate width.1,2 The shared expert captures general principles, while each token independently activates multiple routed experts via sigmoid-based routing for specialized modeling.3,4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
we adopt fine-grained expert segmentation and shared expert isolation, in which a lightweight shared expert preserves universal priors, while a set of routed experts provides specialized modeling capacity
— From Foundation to Application: Improving VLA Models in Practiceboth the shared and routed experts employ a smaller intermediate width
— From Foundation to Application: Improving VLA Models in Practiceemploying sigmoid-based routing confidence to allow each token to independently activate multiple experts
— From Foundation to Application: Improving VLA Models in Practiceencouraging shared experts to capture general principles and routed experts to provide stronger specialization
— From Foundation to Application: Improving VLA Models in Practice
Also called shared and fine-grained routed experts.
Filed under model architecture › channel mixer › mixture of experts › fine-grained experts