Kimi K3
Moonshot AIStable LatentMoE addresses these two failure modes with three components: an RMSNorm before the up-projection and Sigmoid Tanh Unit GLU (SiTU-GLU) to suppress activation explosion, and Quantile Balancing (QB) for load balancing.
6 more from this model
Kimi K3 uses Stable LatentMoE, effectively activating 16 of 896 experts.
Kimi K3 instead inserts RMSNorm [146] between expert aggregation and the up-projection.
The combination of Stable LatentMoE, KDA, and quantization-aware training
scales up MoE sparsity with a Stable LatentMoE framework that activates 16 out of 896 experts
scales up MoE sparsity with a Stable LatentMoE framework that activates 16 out of 896 experts
paired with a Stable LatentMoE framework