Model	filed here	global attention	sliding window attention	grouped-query attention	multi-head latent attention	attention sink
DeepSeek-V4.1-Flash core	—	—	Sliding Window Attention coreDecoder SWA Bounded Replay usedPure Sliding Window Attention used—	—	—	—
NVIDIA-Nemotron-3-Ultra-550B-A55B core	Attention layers core—	—	—	—	—	—
MiMo-V2.6-Flash core	—	—	Alternating Row-Major and Column-Major Token Serialization coreSink-Augmented Sliding Window Attention core—	—	—	—
DeepSeek-V4-Flash used	—	—	Sliding Window Attention used—	Multi-query attention used—	—	Attention sink used—
MiMo-V2.5 core	—	Global attention coreFull attention used—	Hybrid Sliding Window Attention coreSliding Window Attention coreSWA-128 used—	Grouped-query attention used—	—	Learnable attention sink bias used—
Hy3 core	—	—	—	Grouped-query attention core—	—	—
GLM-5.2 core	—	—	Sliding Window Attention evaluated—	Multi-query attention mentioned—	Multi-Head Latent Attention core—	—
MiniMax-M3 core	—	Full attention evaluated—	Fixed Sliding Window Attention evaluatedSliding Window Attention mentioned—	Block-sparse grouped-query attention coreGrouped-query attention core—	Multi-Head Latent Attention not used—	Learnable attention sink bias not usedRoPE with attention sink unclear—
DeepSeek-V3.2 core	—	—	—	—	MQA Mode of Multi-Head Latent Attention coreMulti-Head Latent Attention MHA/MQA Modes used—	—
DeepSeek-V4-Flash-Vision-Exp core	DFlash attention core—	—	—	—	—	—
DeepSeek-V4-Pro used	—	—	Sliding Window Attention used—	Multi-query attention used—	—	Attention sink used—
Gemma 4 31B core	—	Unified Keys and Values core—	—	—	—	—
Inkling used	—	—	512-token Sliding Window Attention used—	Grouped-query attention used—	—	—
Kimi K3 core	—	—	—	—	Gated Multi-Head Latent Attention core—	—
Laguna-S-2.1 core	Softplus attention gating core—	—	512-token Sliding Window Attention evaluatedPer-Head Gating evaluatedSWA-1024 evaluated—	Grouped-query attention coreSoftplus-based per-head gating used—	—	—
MiMo-V2.5-Pro used	—	Global attention used—	Hybrid Sliding Window Attention usedSliding Window Attention used—	—	—	Learnable attention sink bias used—
MiMo-V2.6-Pro core	—	—	Alternating Row-Major and Column-Major Token Serialization coreSink-Augmented Sliding Window Attention core—	—	—	—
Qwen3.5-397B-A17B used	—	Multi-Head Attention (MHA) used—	Dynamic Attention Window Size Training used—	—	—	—
Step-3.7-Flash not used	Cascade attention not used—	—	—	—	—	—
gpt-oss-120b used	Learned softmax denominator bias used—	—	—	Grouped-query attention used—	—	—

Model

filed here

global attention

sliding window attention

grouped-query attention

multi-head latent attention

attention sink

DeepSeek-V4.1-Flash core

—

Sliding Window Attention coreDecoder SWA Bounded Replay usedPure Sliding Window Attention used—

—

NVIDIA-Nemotron-3-Ultra-550B-A55B core