specific method · filed under model architecture
Hybrid Sliding Window Attention
Hybrid sliding window attention is an attention architecture that interleaves sliding window attention with global attention.1 In sliding window attention, each token attends only to a local window instead of the whole sequence, which reduces computational and memory complexity.2
Trade-off Overly aggressive use of sliding window attention can substantially degrade model performance, especially on long-context tasks.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
hybrid attention architectures that interleave sliding window attention with global attention
— MiMo-V2-Flash Technical Reportrestricts each token’s attention scope to a local window rather than the entire sequence, thereby reducing both computational and memory complexity
— MiMo-V2-Flash Technical Reportoverly aggressive use of SWA, such as very small sliding window sizes or high SWA:GA ratios, can lead to substantial degradation in model performance
— MiMo-V2-Flash Technical Report
Also called Hybrid window attention.