All deployment snippets below default to port 8000, with chunked prefill, NVFP4 KV caching, and MTP (5 speculative tokens) enabled
1 more from this model
enable-chunked-prefill
specific method · filed under inference & serving
Chunked prefill gives each scheduler step a token budget, fills it with the running decodes first and then with a slice of a pending prompt, so a long prompt is spread over many steps. The budget trades time to first token against the time per decode step, and each chunk re-reads the weights and the cache written so far.
Written for the explainer Disaggregation and the KV-cache system: scheduling, prefix reuse and tiered memory.
Also called --chunked-prefill-size 16384, chunked prefilling, chunked-prefilling mechanism.
6 quotes from 6 sources, one card per model, each opening with its strongest.
All deployment snippets below default to port 8000, with chunked prefill, NVFP4 KV caching, and MTP (5 speculative tokens) enabled
enable-chunked-prefill
--chunked-prefill-size 16384
--chunked-prefill-size 16384
--enable-chunked-prefill
we retain the chunked-prefilling mechanism as implemented in Qwen3-Omni and Qwen2.5-Omni, whose audio and vision encoders are capable of outputting chunks along the temporal dimension.
Picked by hand, not extracted: where to read more, not evidence for anything on this page.