Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertslatent mixture of experts

specific method · filed under model architecture

LatentMoE

LatentMoE is a mixture-of-experts architecture that routes each token's embedding into a latent representation of smaller dimension before expert computation.1 The savings in weights and communication are reinvested in more experts and more active experts per token, raising quality at a similar budget.2

Trade-off Shrinking the routed expert dimension cuts per-expert weight loads and communication, offsetting the larger expert count so quality rises without extra compute or bandwidth.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Each token embedding is first projected from the original hidden dimension into a latent representation of smaller dimension , routed to an expanded set of experts that operate entirely in this latent space — NVIDIA Nemotron 3: Efficient and Open Intelligence
  2. We use these parameter and bandwidth savings to increase both the total number of experts from to and the top- active experts per token from to — NVIDIA Nemotron 3: Efficient and Open Intelligence
  3. By shifting routed expert computation and all-to-all traffic to the latent space, both per-expert weight loads and communication payloads are reduced by a factor of compared to a standard MoE — NVIDIA Nemotron 3: Efficient and Open Intelligence

Also called Latent Mixture of Experts, Latent Mixture-of-Experts (LatentMoE), Latent MoE, LatentMixture of Experts.

sources
7
models
3
labs adopt it
2
strongest
core

Documented in

Evidence

9 quotes from 7 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 8

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted5 sources

LatentMoE (Elango et al., 2026), which buys more routed experts at fixed inference cost

coremodel architecturein Nemotron 3 UltraNVIDIA
6 more from this model

LatentMoE helped us achieve better accuracy per parameter than standard Granular MoEs

coremodel architecturein Nemotron 3 UltraNVIDIA

LatentMoE (latentmoe_tr), which buys more routed experts at fixed inference cost

coremodel architecturein Nemotron 3 UltraNVIDIA

The model employs a hybrid Latent Mixture-of-Experts (LatentMoE) architecture

coremodel architecturein Nemotron 3 UltraNVIDIA

The model employs a hybrid Latent Mixture-of-Experts (LatentMoE) architecture

coremodel architecturein Nemotron 3 UltraNVIDIA

LatentMoE (Elango et al., 2026) for MoE layers

coremodel architecturein Nemotron 3 UltraNVIDIA

Leverages LatentMoE for improved accuracy.

usedmodel architecturein Nemotron 3 UltraNVIDIA

Kimi K3

Moonshot AIadopted1 source

LatentMoE [32] makes this expansion affordable by separating the full model width from the routed-expert width

coremodel architecturein Kimi K3Moonshot AI

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑