Additional Mamba checkpoints reduce the replay needed to recover inputs for speculative decoding
1 more from this model
We improved this path by optionally retaining an additional Mamba cache page near the speculative-decoding replay boundary.
implementation detail · filed under inference & serving
Retains additional Mamba cache checkpoints to reduce replay needed for speculative-decoding recovery.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See KV cache management for the methods filed beside it.
Also called Additional Mamba checkpoints.
2 quotes from 2 sources, one card per model, each opening with its strongest.
Additional Mamba checkpoints reduce the replay needed to recover inputs for speculative decoding
We improved this path by optionally retaining an additional Mamba cache page near the speculative-decoding replay boundary.
We improved this path by optionally retaining an additional Mamba cache page near the speculative-decoding replay boundary.