Open Models Almanac
Methodsinference & servingKV cache management

specific method · filed under inference & serving

Chunked prefill

Chunked prefill gives each scheduler step a token budget, fills it with the running decodes first and then with a slice of a pending prompt, so a long prompt is spread over many steps. The budget trades time to first token against the time per decode step, and each chunk re-reads the weights and the cache written so far.

Written for the explainer Disaggregation and the KV-cache system: scheduling, prefix reuse and tiered memory.

Also called --chunked-prefill-size 16384, chunked prefilling, chunked-prefilling mechanism.

sources
6
models
4
labs adopt it
3
strongest
default

Documented in

Evidence

6 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
optional 1used 3default 2

Qwen3.5-397B-A17B

Alibaba (Qwen)adopted1 source

we retain the chunked-prefilling mechanism as implemented in Qwen3-Omni and Qwen2.5-Omni, whose audio and vision encoders are capable of outputting chunks along the temporal dimension.

usedinference servingin Qwen3.5-OmniQwen Team

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑