Open Models Almanac
Methodsinference & servinginference kernel

single mention · inference & serving

Dual-kernel batch-invariant attention decoding

A dual-kernel strategy that makes attention decoding batch-invariant, so a token's output is bitwise identical regardless of its position within a batch.1,2 A first kernel computes a whole sequence's attention on a single SM, while a second uses multiple SMs for the final partially-filled wave, matching the first kernel's accumulation order for bitwise identity.3,4,5

Trade-off The dual-kernel design keeps the overhead of batch-invariant decoding negligible.6

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we develop a dual-kernel strategy for batch-invariant decoding — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. Batch invariance ensures that the output of any given token remains bitwise identical, regardless of its position within a batch — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  3. The first kernel computes the attention output for an entire sequence within a single SM — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  4. The second kernel, to minimize the latency of the final partially-filled wave and thus alleviate wave-quantization, uses multiple SMs for a single sequence — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  5. we carefully design the calculation path of the second kernel to ensure its accumulation order is the same as that of the first kernel — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  6. This dual-kernel method effectively confines the overhead of batch-invariant decoding to be negligible — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Filed under inference & serving › inference kernel