Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionsliding window attention

single mention · model architecture

Decoder SWA Bounded Replay

Decoder SWA Bounded Replay bounds the decoder's prefill replay to the last window-size tokens of the prompt when regenerating sliding-window attention KV.1,2 It avoids the expensive full decoder pass over window×layer tokens, since uncached decoder SWA KV must otherwise be recomputed, and the approximate replayed KV is used only for decoding.3,4

Trade-off The replayed SWA KV is only approximate, but experiments show this barely compromises response quality.5

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. only prefills the last𝑛win tokens of the prompt for the SWA computation — DeepSeek V41 Tech Report
  2. bounds the decoder forward pass to𝑛 win tokens — DeepSeek V41 Tech Report
  3. exactly reconstructing it requires running the𝐿 2 decoder layers over the last 𝐿 2×𝑛 win prompt tokens, which is expensive — DeepSeek V41 Tech Report
  4. use the resulting decoder SWA KV only for decoding, not for prefix caching — DeepSeek V41 Tech Report
  5. the replayed prefix state is approximate — DeepSeek V41 Tech Report

Also called decoder bounded replay during prefill.

Filed under model architecture › token mixer › softmax attention › sliding window attention