Open Models Almanac
Methodsinference & servingserving parallelism

single source · inference & serving

Zero-copy fused token permutation and unpermutation

It sends tokens directly to their expert-grouped positions on remote ranks, eliminating intermediate copies.1 so tokens are sent directly to their expert-grouped positions on remote ranks, and views of the communication buffer are returned directly to the computation, eliminating intermediate copies.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We implement a fused per- mute/unpermute operator in which the planning kernel precomputes the destination of every token — k3 tech report
  2. so tokens are sent directly to their expert-grouped positions on remote ranks, and views of the communication buffer are returned directly to the computation, eliminating intermediate copies — k3 tech report

Also called zero-copy communication, fused per-mute/unpermute operator.

Filed under inference & serving › serving parallelism