single mention · inference & serving
Dual-kernel batch-invariant attention decoding
A dual-kernel strategy that makes attention decoding batch-invariant, so a token's output is bitwise identical regardless of its position within a batch.1,2 A first kernel computes a whole sequence's attention on a single SM, while a second uses multiple SMs for the final partially-filled wave, matching the first kernel's accumulation order for bitwise identity.3,4,5
Trade-off The dual-kernel design keeps the overhead of batch-invariant decoding negligible.6
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
we develop a dual-kernel strategy for batch-invariant decoding
— DeepSeek-V4: Towards Highly Efficient Million-Token Context IntelligenceBatch invariance ensures that the output of any given token remains bitwise identical, regardless of its position within a batch
— DeepSeek-V4: Towards Highly Efficient Million-Token Context IntelligenceThe first kernel computes the attention output for an entire sequence within a single SM
— DeepSeek-V4: Towards Highly Efficient Million-Token Context IntelligenceThe second kernel, to minimize the latency of the final partially-filled wave and thus alleviate wave-quantization, uses multiple SMs for a single sequence
— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencewe carefully design the calculation path of the second kernel to ensure its accumulation order is the same as that of the first kernel
— DeepSeek-V4: Towards Highly Efficient Million-Token Context IntelligenceThis dual-kernel method effectively confines the overhead of batch-invariant decoding to be negligible
— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Filed under inference & serving › inference kernel