specific method · filed under model architecture
MoE with Expert Sinks
A configuration where shared experts contribute to routing-score computation but are excluded from the routed experts' top-k candidate pool.
Also called Mixture-of-Experts with expert sinks.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
core 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
The 2 shared experts act as expert sinks: they participate in routing-score computation but are excluded from the top-6 candidate pool.
coremodel architecturein InklingThinking Machines Lab
Filed alongside
Other methods under model architecture :: channel mixer :: mixture of experts :: shared experts.