single mention · inference & serving
Sparse pinned-host offload
Sparse pinned-host offload keeps each rank's vocabulary-parallel table shard in pinned host memory.1 It gathers the selected rows into a small BF16 GPU buffer using a Triton UVA kernel.2
Trade-off The offload reduces target-model weights and increases allocated KV capacity.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
SGLang keeps each rank's vocabulary-parallel table shard in pinned host memory
— Qwen3.8-Flash-Next: Day-0 Support in SGLanggathers the selected rows into a small BF16 GPU buffer with a Triton UVA kernel
— Qwen3.8-Flash-Next: Day-0 Support in SGLangoffload reduced target-model weights from 83.91 to 60.45 GiB per GPU (-23.46 GiB) and increased allocated KV capacity from 1.84M to 3.28M tokens (+78.54%) at the same memory fraction
— Qwen3.8-Flash-Next: Day-0 Support in SGLang
Filed under inference & serving › inference kernel