Hybrid Attention

Remaining blocks interleave local SWA and GA

Its hybrid sparse and linear attention architecture maintains accurate long-context behavior while reducing compute overhead.

repeated hybrid blocks that interleave Local Sliding Window Attention (SWA) and Global Attention (GA)

a Transformer with a causal hybrid attention architecture that interleaves SWA and GA layers

MiMo-V2.6 interleaves 128-token sliding-window layers with full attention.

The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global.

Attention is a hybrid of local and global layers.

Kimi K3 uses a layerwise hybrid of linear and global attention, combining KDA [63] with Gated MLA.

For the first time in the GLM series, Z.ai combines linear and sparse attention.

The 3:1 interleave of Gated DeltaNet to full-attention layers is the same ratio Qwen3.5 uses

we introduce a hybrid architecture combining sparse and linear attention

Hybrid Attention Architecture: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 6:1 ratio and 128 sliding window.

Attention is a hybrid of local and global layers.

Attention is a hybrid of local and global layers

Hybrid Attention Architecture: Inherits the hybrid design from MiMo-V2-Flash, interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and 128 sliding window.

it employs a hybrid attention mechanism for greater computational efficiency

The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global.

The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global.

Mixed SWA and global attention layout: 48 layers in a 1:3 global-to-SWA ratio (12 global attention layers, 36 sliding-window layers, window 512)

Hybrid Attention Architecture: Inherits the hybrid design from MiMo-V2-Flash, interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and 128 sliding window.

MiMo-V2-Flash adopts a hybrid attention architecture that interleaves Sliding Window Attention (SWA) with global attention

hybrid attention (SWA-128 + full-attention at 5:1 ratio)

introducing several key architectural innovations: (1) Hybrid Attention Architecture: combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency.

attention blocks alternate between banded window and fully dense patterns

We conserve a 5:1 ratio of local sliding window to global self-attention (4:1 for the 2.3B model)

For attention, we interleave sliding-window and global layers at a 5:1 ratio with 8 KV heads.

hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

LAGUNA XS.2 uses interleaved Sliding Window Attention (SWA) and Global Attention (GA) with a 3:1 ratio.

Hybrid Attention Architecture: We design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency.