Open Models Almanac
Methodsinference & servinginference scheduling

specific method · filed under inference & serving

Session affinity routing

Sends turns from the same session to the same worker so later turns can reuse its prefix cache.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See inference scheduling for the methods filed beside it.

Also called Session affinity.

sources
2
models
2
lab adopt it
1
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1used 1

Session affinity sends all turns of one session to the same worker, so later turns find their prefix in its cache.

usedinference servingin Nemotron 3.5 Super VLNVIDIA
1 more from this model

KV-aware routing with session affinity and KV cache offload serves up to 8x as many users

evaluatedinference servingin NVIDIA DynamoNVIDIA
↑