Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert routing

specific method · filed under model architecture

Anticipatory Routing

Anticipatory Routing is a MoE routing method that decouples the synchronous updates of the backbone network and the routing network to improve training stability.1 At each step, features are computed with the current parameters, while routing indices are computed with historical parameters and cached in advance for later use.2,3

Trade-off It averts loss spikes without compromising model performance, at the cost of roughly 20% additional wall-clock time while active.4,5

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. decoupling the synchronous updates of the backbone network and the routing network significantly improves training stability — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. the routing indices are computed and applied using the historical network parameters — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  3. We "anticipatorily" compute and cache the routing indices to be used later at step — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  4. bounding the additional wall-clock time overhead of Anticipatory Routing to approximately 20 — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  5. avert loss spikes with negligible overall additional training overhead, all without compromising model performance — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
sources
2
models
2
lab adopt it
1
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
optional 1used 1

DeepSeek-V4-Pro

DeepSeekadopted2 sources

we "anticipatorily" compute and cache the routing indices to be used later at step

optionaloptimizationin DeepSeek-V4DeepSeek
1 more from this model

Anticipatory Routing (computing routing indices one step ahead to decouple backbone and routing updates)

usedtraining objectivein DeepSeek-V4DeepSeek
↑