specific method · filed under model architecture
LatentMoE
LatentMoE is a mixture-of-experts architecture that routes each token's embedding into a latent representation of smaller dimension before expert computation.1 The savings in weights and communication are reinvested in more experts and more active experts per token, raising quality at a similar budget.2
Trade-off Shrinking the routed expert dimension cuts per-expert weight loads and communication, offsetting the larger expert count so quality rises without extra compute or bandwidth.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Each token embedding is first projected from the original hidden dimension into a latent representation of smaller dimension , routed to an expanded set of experts that operate entirely in this latent space
— NVIDIA Nemotron 3: Efficient and Open IntelligenceWe use these parameter and bandwidth savings to increase both the total number of experts from to and the top- active experts per token from to
— NVIDIA Nemotron 3: Efficient and Open IntelligenceBy shifting routed expert computation and all-to-all traffic to the latent space, both per-expert weight loads and communication payloads are reduced by a factor of compared to a standard MoE
— NVIDIA Nemotron 3: Efficient and Open Intelligence
Also called Latent Mixture of Experts, Latent Mixture-of-Experts (LatentMoE), Latent MoE, LatentMixture of Experts.