Open Models Almanac
Methodsinference & servinginference kernel

single mention · inference & serving

Sparse pinned-host offload

Sparse pinned-host offload keeps each rank's vocabulary-parallel table shard in pinned host memory.1 It gathers the selected rows into a small BF16 GPU buffer using a Triton UVA kernel.2

Trade-off The offload reduces target-model weights and increases allocated KV capacity.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. SGLang keeps each rank's vocabulary-parallel table shard in pinned host memory — Qwen3.8-Flash-Next: Day-0 Support in SGLang
  2. gathers the selected rows into a small BF16 GPU buffer with a Triton UVA kernel — Qwen3.8-Flash-Next: Day-0 Support in SGLang
  3. offload reduced target-model weights from 83.91 to 60.45 GiB per GPU (-23.46 GiB) and increased allocated KV capacity from 1.84M to 3.28M tokens (+78.54%) at the same memory fraction — Qwen3.8-Flash-Next: Day-0 Support in SGLang

Filed under inference & serving › inference kernel