specific method · filed under model architecture
Normalized LatentMoE
A LatentMoE variant that adds RMSNorm before the up-projection and uses SiTU-GLU and Quantile Balancing to address activation instability and load balancing.
Also called Stable LatentMoE, Stable LatentMoE framework.
- sources
- 6
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Kimi K3: Open Frontier Intelligence (Kimi Team) paper arxiv.orgintroduces Stable LatentMoE
Evidence
7 spans quoted from the sources, strongest treatment first.
scales up MoE sparsity with a Stable LatentMoE framework that activates 16 out of 896 experts
Stable LatentMoE addresses these two failure modes with three components: an RMSNorm before the up-projection and Sigmoid Tanh Unit GLU (SiTU-GLU) to suppress activation explosion, and Quantile Balancing (QB) for load balancing.
The combination of Stable LatentMoE, KDA, and quantization-aware training
Kimi K3 uses Stable LatentMoE, effectively activating 16 of 896 experts.
scales up MoE sparsity with a Stable LatentMoE framework that activates 16 out of 896 experts
paired with a Stable LatentMoE framework
Kimi K3 instead inserts RMSNorm [146] between expert aggregation and the up-projection.
Filed alongside
Other methods under model architecture :: channel mixer :: mixture of experts :: latent mixture of experts.