Session affinity sends all turns of one session to the same worker, so later turns find their prefix in its cache.
1 more from this model
KV-aware routing with session affinity and KV cache offload serves up to 8x as many users
specific method · filed under inference & serving
Sends turns from the same session to the same worker so later turns can reuse its prefix cache.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See inference scheduling for the methods filed beside it.
Also called Session affinity.
2 quotes from 2 sources, one card per model, each opening with its strongest.
Session affinity sends all turns of one session to the same worker, so later turns find their prefix in its cache.
KV-aware routing with session affinity and KV cache offload serves up to 8x as many users
Session affinity sends all turns of one session to the same worker, so later turns find their prefix in its cache.