single mention · inference & serving
Concurrency-aware draft-length tuning
Adjusts speculative draft length in response to concurrency, with the reported optimal length decreasing as concurrency increases.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See decoding strategy for the methods filed beside it.
Filed under inference & serving › decoding strategy