implementation detail · filed under inference & serving
Zero-copy fused token permutation and unpermutation
Fuses token permutation and unpermutation, sending tokens directly to expert-grouped positions and returning communication-buffer views without intermediate copies.
Also called zero-copy communication, fused per-mute/unpermute operator.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
We implement a fused per-mute/unpermute operator in which the planning kernel precomputes the destination of every token, so tokens are sent directly to their expert-grouped positions on remote ranks, and views of the communication buffer are returned directly to the computation, eliminating intermediate copies.
We implement a fused per-mute/unpermute operator in which the planning kernel precomputes the destination of every token, so tokens are sent directly to their expert-grouped positions on remote ranks, and views of the communication buffer are returned directly to the computation, eliminating intermediate copies.
Filed alongside
Other methods under inference & serving :: serving parallelism.