Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertslatent mixture of experts

specific method · filed under model architecture

Stable LatentMoE

Stable LatentMoE is a latent mixture-of-experts framework that separates full-width shared experts from routed experts operating in a compact latent space, enabling a large expert pool with few active experts per token.1,2 It adds stabilization components — RMSNorm before the up-projection, SiTU-GLU, and Quantile Balancing — to address activation explosion and load-balancing failures that extreme sparsity amplifies.3

Trade-off The extreme sparsity it enables amplifies two failure modes of the vanilla design: exploding internal activations and load balancing beyond existing methods' regime.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. shared experts retain a full-width path for common transformations, whereas specialized routed experts operate in a compact latent space of width ℓ — k3 tech report
  2. This enables Kimi K3 to scale channel mixing to 896 routed experts with 16 active experts per token — k3 tech report
  3. Stable LatentMoE addresses these two failure modes with three components: an RMSNorm before the up-projection and Sigmoid Tanh Unit GLU (SiTU-GLU) to suppress activation explosion, and Quantile Balancing (QB) for load balancing — k3 tech report
  4. This extreme sparsity amplifies two failure modes of the vanilla design — k3 tech report

Also called Stable LatentMoE framework, Normalized LatentMoE.

sources
6
model
1
lab adopt it
1
strongest
core

Documented in

Moonshot AI

Evidence

7 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 5

Kimi K3

Moonshot AIadopted6 sources

Stable LatentMoE addresses these two failure modes with three components: an RMSNorm before the up-projection and Sigmoid Tanh Unit GLU (SiTU-GLU) to suppress activation explosion, and Quantile Balancing (QB) for load balancing.

coremodel architecturein Kimi K3Moonshot AI
6 more from this model

Kimi K3 uses Stable LatentMoE, effectively activating 16 of 896 experts.

coremodel architecturein Kimi K3Moonshot AI

Kimi K3 instead inserts RMSNorm [146] between expert aggregation and the up-projection.

usedmodel architecturein Kimi K3Moonshot AI

The combination of Stable LatentMoE, KDA, and quantization-aware training

coremodel architecturein Kimi K3Moonshot AI

scales up MoE sparsity with a Stable LatentMoE framework that activates 16 out of 896 experts

coremodel architecturein Kimi K3Moonshot AI

scales up MoE sparsity with a Stable LatentMoE framework that activates 16 out of 896 experts

coremodel architecturein Kimi K3Moonshot AI

paired with a Stable LatentMoE framework

usedmodel architecturein Kimi K3Moonshot AI

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑