single mention · model architecture
Sigmoid-based expert routing
Sigmoid-based expert routing is a token-to-expert routing scheme for mixture-of-experts layers that scores each token's affinity to experts with a sigmoid instead of softmax.1 It is motivated by avoiding the strong competition among experts that softmax normalization induces.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
we apply a sigmoid-activated function to the router logits
— From Foundation to Application: Improving VLA Models in PracticeTo avoid the strong competition among experts induced by softmax normalization
— From Foundation to Application: Improving VLA Models in Practice
Also called Sigmoid-based token-to-expert routing.
Filed under model architecture › channel mixer › mixture of experts › expert routing