Hy4-preview
Tencent (Hunyuan)every token activates the top-8 routed experts along with the shared expert
1 more from this model
top-8
implementation detail · filed under model architecture
The sources on this page only mention it in passing. See expert routing for the methods filed beside it.
12 quotes from 11 sources, one card per model, each opening with its strongest.
every token activates the top-8 routed experts along with the shared expert
top-8
each token is routed to 6 of 256 experts, plus 2 shared experts active on every token
each token is routed to 6 of 256 experts
Number of Experts | 192 experts, top-8 activated
Hy3’s architecture contains a sparse MoE with 192 experts and top-8 routing.
a softmax router selects four of the 128 experts for each token.
each MoE layer routes a token to 6 of its 384 routed experts
routes each token through 8 of 288 experts
Each MoE layer uses 128 routed experts, 1 shared expert, and top-4 routed expert selection.
top-4 expert routing
it uses 256 routed experts (top-8)