implementation detail · filed under model architecture
Decoder SWA Bounded Replay
A prefill procedure that replays or prefills only the last nwin prompt tokens for decoder computation under sliding-window truncation.
Also called decoder bounded replay during prefill.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 2
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
we introduce Decoder SWA Bounded Replay, which only prefills the last nwin tokens of the prompt for the SWA computation
usedinference servingin DeepSeek-V4.1-FlashDeepSeek
at every prefill, we replay the last 𝑛win tokens of the prompt, feed their encoder outputs through the decoder layers under the same SWA truncation
usedunclearin DeepSeek-V4.1-FlashDeepSeek
Filed alongside
Other methods under model architecture :: token mixer :: softmax attention :: sliding window attention.
Sliding Window AttentionHybrid Sliding Window Attention512-token Sliding Window AttentionAlternating Row-Major and Column-Major Token SerializationDynamic Attention Window Size TrainingFixed Sliding Window AttentionPer-Head GatingPure Sliding Window AttentionSink-Augmented Sliding Window AttentionSWA-1024SWA-128