Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionsliding window attention

specific method · filed under model architecture

Hybrid Sliding Window Attention

Hybrid sliding window attention is an attention architecture that interleaves sliding window attention with global attention.1 In sliding window attention, each token attends only to a local window instead of the whole sequence, which reduces computational and memory complexity.2

Trade-off Overly aggressive use of sliding window attention can substantially degrade model performance, especially on long-context tasks.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. hybrid attention architectures that interleave sliding window attention with global attention — MiMo-V2-Flash Technical Report
  2. restricts each token’s attention scope to a local window rather than the entire sequence, thereby reducing both computational and memory complexity — MiMo-V2-Flash Technical Report
  3. overly aggressive use of SWA, such as very small sliding window sizes or high SWA:GA ratios, can lead to substantial degradation in model performance — MiMo-V2-Flash Technical Report

Also called Hybrid window attention.

sources
4
models
2
lab adopt it
1
strongest
core

Documented in

Evidence

5 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
used 4core 1

MiMo-V2.5

Xiaomiadopted4 sources

through its hybrid Sliding Window Attention architecture, lightweight Multi-Token Prediction, and the MOPD post-training paradigm.

coremodel architecturein MiMo-V2-FlashXiaomi
4 more from this model

We train a dedicated MiMo ViT that adopts sliding-window attention to enable efficient visual encoding.

usedmodel architecturein MiMo-V2.5Xiaomi

Its language backbone inherits from MiMo-V2-Flash's hybrid sliding-window attention architecture

usedmodel architecturein MiMo-V2.5Xiaomi

Equipped with a 729M-param Vision Transformer (ViT) featuring hybrid window attention

usedmodel architecturein MiMo-V2.5Xiaomi

featuring hybrid window attention

usedmodel architecturein MiMo-V2.5Xiaomi

MiMo-V2.5-Pro

Xiaomiadopted1 source

Its language backbone inherits from MiMo-V2-Flash's hybrid sliding-window attention architecture

usedmodel architecturein MiMo-V2.5Xiaomi

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑