implementation detail · filed under model architecture
Dispatch recomputation
A memory-efficient MoE backward-pass technique that recovers the group-GEMM input by recomputing dispatch.
Also called recomputing dispatch.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
core 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
during the backward pass, the input of the group GEMM is recovered by recomputing dispatch.
coresoftware implementationin Kimi K3Moonshot AI
Filed alongside
Other methods under model architecture :: channel mixer :: mixture of experts.
Mixture of ExpertsSparse expert activationDeepSeekMoEGated DeltaNet MoEMixture-of-Experts layersMoE with routed and shared expertsAsymmetric input/output activation splitFrequency-based partitioning of N-gram embedding slotsHybrid Mixture of ExpertsMegaMoEMoE with 128 routed experts and a shared expertMoE with 256 experts and top-8 routingMoE with 384 routed experts and a shared expertOutput-independent MoE gradient reformulationRouted expert output modulationSigmoid-based MoE router with auxiliary-loss-free load balancing