Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Model	filed here	softmax attention	sparse attention	linear attention & state space	hybrid layer stacking
GLM-5.3-Flash core	—	—	IndexPool coreNoPE sparse multi-head latent attention core—	Kimi Delta Attention core—	Hybrid Attention core—
DeepSeek-V4.1-Flash core	Cross-layer attention sharing coreMicro-batch-level shared-state lifetime management used—	Sliding Window Attention coreDecoder SWA Bounded Replay usedPure Sliding Window Attention used—	Compressed Sparse Attention 2 coreCross-layer KV and index reuse with statically assigned CSA2 modes coreHierarchical Sparse Indexer coreReindex Mode coreReuse Mode coreCross-stage shared-state management for attention reuse usedCSA2 Full Mode usedFrom-scratch sparse attention training without dense warmup usedProgressive sequence-length extension for sparse attention usedSparse retrieval over long contexts evaluated—	—	—
Hy4-preview core	—	—	Gated DeepSeek Sparse Attention coreIndexCache core—	—	—
NVIDIA-Nemotron-3-Ultra-550B-A55B core	—	Attention layers core—	Sparse Global Attention Anchors core—	Mamba coreMamba-2 coreMamba-2 SSM cache optional—	Hybrid Mamba-Attention coreHybrid Mamba-Transformer core—
MiMo-V2.6-Flash core	—	Alternating Row-Major and Column-Major Token Serialization coreSink-Augmented Sliding Window Attention core—	—	—	Hybrid Attention coreHybrid sparse mixture-of-experts Transformer architecture core—
DeepSeek-V4-Flash core	—	Attention sink usedMulti-query attention usedSliding Window Attention used—	Compressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreTwo-stage introduction of sparse attention used—	—	Hybrid Attention used—
MiMo-V2.5 core	—	Global attention coreHybrid Sliding Window Attention coreSliding Window Attention coreFull attention usedGrouped-query attention usedLearnable attention sink bias usedSWA-128 used—	—	—	Hybrid Attention core—
Hy3 core	—	Grouped-query attention core—	—	—	—
GLM-5.2 core	—	Multi-Head Latent Attention coreSliding Window Attention evaluatedMulti-query attention mentioned—	DeepSeek Sparse Attention coreIndexShare coreLightning Indexer usedSparse attention used—	Gated DeltaNet evaluatedSimpleGDN evaluated—	Search-Based Sliding-Window Attention Pattern evaluated—
MiniMax-M3 core	—	Block-sparse grouped-query attention coreGrouped-query attention coreFixed Sliding Window Attention evaluatedFull attention evaluatedSliding Window Attention mentionedLearnable attention sink bias not usedMulti-Head Latent Attention not usedRoPE with attention sink unclear—	Block-level selection coreDynamic sparse selection coreIndex Branch coreMain Branch coreMiniMax Sparse Attention coreSequential block processing coreSparse attention coreBlock Max Pooling usedIndexer Warmup usedKV-outer sparse attention usedLocal Block usedNatively trained sparsity usedSingle-head index key usedSparse softmax attention usedTop-k block selection usedTwo-stage sparse attention usedDeepSeek Sparse Attention evaluatedMixture of Block Attention evaluatedCompressed Sparse Attention not usedIndex Branch output not usedIndex Branch value head not usedNative Sparse Attention not used—	Linear attention mentionedLightning Attention not used—	Dense Attention Fallback unclear—
DeepSeek-V3.2 core	—	MQA Mode of Multi-Head Latent Attention coreMulti-Head Latent Attention MHA/MQA Modes used—	DeepSeek Sparse Attention coreFine-grained token selection coreLightning Indexer coreDense Warm-up Stage usedDetached indexer-input optimization usedSparse-attention continued pre-training with joint model and indexer optimization used—	—	—
DeepSeek-V4-Flash-Vision-Exp core	—	DFlash attention core—	—	—	—
DeepSeek-V4-Pro core	—	Attention sink usedMulti-query attention usedSliding Window Attention used—	Compressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreTwo-stage introduction of sparse attention used—	—	Hybrid Attention used—
Gemma 4 31B core	—	Unified Keys and Values core—	—	—	Hybrid Attention core—
Inkling core	Kernel-4 convolutional layers in decoder blocks used—	512-token Sliding Window Attention usedGrouped-query attention used—	—	—	Hybrid Attention core—
Kimi K3 core	Factorized spatial-temporal video attention with temporal pooling used—	Gated Multi-Head Latent Attention core—	—	Hybrid linear attention coreKDA coreKimi Delta Attention coreKimi Delta Attention and Attention Residuals architecture coreGated DeltaNet usedKimi Delta Attention with input-dependent full-rank output gate usedKimi Delta Attention with lower-bounded log-decay used—	Hybrid Attention core—
Laguna-S-2.1 core	Dense gated attention with full RoPE and full gating evaluated—	Grouped-query attention coreSoftplus attention gating coreSoftplus-based per-head gating used512-token Sliding Window Attention evaluatedPer-Head Gating evaluatedSWA-1024 evaluated—	—	—	Hybrid Attention coreGated Attention and Sliding-Window Attention Configuration used—
MiMo-V2.5-Pro core	—	Global attention usedHybrid Sliding Window Attention usedLearnable attention sink bias usedSliding Window Attention used—	—	—	Hybrid Attention core—
MiMo-V2.6-Pro core	—	Alternating Row-Major and Column-Major Token Serialization coreSink-Augmented Sliding Window Attention core—	—	—	Hybrid Attention coreHybrid sparse mixture-of-experts Transformer architecture core—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B core	—	—	—	Mamba-2 core—	Hybrid Mamba-Transformer coreMamba-2, Mixture-of-Experts, and Selective Attention Hybrid core—
Qwen3.5-397B-A17B core	—	Dynamic Attention Window Size Training usedMulti-Head Attention (MHA) used—	DeepSeek Sparse Attention core—	Gated DeltaNet coreGated DeltaNet with reduced KV-head configuration coreGated DeltaNet–sparse MoE hybrid coreLinear attention core—	Gated DeltaNet and Full Attention corehybrid Gated DeltaNet + sparse MoE architecture coreHybrid Mamba-Transformer Mixture-of-Experts Layer Layout core—
Qwen3.6-35B-A3B core	—	—	—	Gated DeltaNet core—	Gated DeltaNet and Gated Attention core—
Qwen3.8-Flash-Next core	—	—	Compressed lightweight indexer coreFixed-budget sparse-attention selection coreMQA indexer coreQSA micro-block compression at ratio 4 coreQwen Sparse Attention coreReuse QSA index selection across speculative decoding steps coreTop-512 block selection coreAverage pooling usedBlock-causal scoring usedJoint backbone and indexer training under sparse attention usedTop-scoring compressed-block selection used—	Gated DeltaNet coreGated DeltaNet with bounded sigmoid output gate used—	Gated DeltaNet and Qwen Sparse Attention coreHybrid Attention core—
Step-3.7-Flash not used	—	Cascade attention not used—	—	—	—
gpt-oss-120b used	—	Grouped-query attention usedLearned softmax denominator bias used—	—	—	Hybrid Attention used—