single mention · model architecture
Output-independent MoE gradient reformulation
A gradient transformation that removes the backward dependency on the output by expressing the gradient in terms of intermediate activations and the upstream gradient.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See mixture of experts for the methods filed beside it.
Filed under model architecture › channel mixer › mixture of experts