implementation detail · filed under inference & serving
Language-model-only serving mode
Skips the vision encoder and multimodal profiling to free memory for KV caching.
Also called language-model-only, Text-Only.
- sources
- 3
- models
- 2
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
3 spans quoted from the sources, strongest treatment first.
use --language-model-only to skip loading the vision encoder and free up memory for KV cache, and enable Expert Parallelism.
The following command skips the vision encoder and multimodal profiling to free up memory for additional KV cache
The following command skips the vision encoder and multimodal profiling to free up memory for additional KV cache: - vllm serve ... --language-model-only
Filed alongside
Other methods under inference & serving :: KV cache management.