Model techniques map
Taxonomymodel architecturetoken mixerhybrid layer stacking

taxonomy node · level 3

hybrid layer stacking

13 methods filed at this node or below it, from the sources of 19 models.

model architecture :: token mixer :: hybrid layer stacking

Matching aids for the classifier: hybrid attention architecture; interleaved SWA and global; hybrid Mamba-Transformer; dense attention fallback.

In this branch 13

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 13

Hybrid Attention core · 27 sources · 29 quotes
Hybrid Mamba-Transformer core · 4 sources · 6 quotes
Mamba-2, Mixture-of-Experts, and Selective Attention Hybrid core · 4 sources · 7 quotes
Hybrid Mamba-Attention core · 3 sources · 4 quotes
Hybrid Mamba-Transformer Mixture-of-Experts Layer Layout core · 2 sources · 2 quotes
Gated DeltaNet and Full Attention core · 1 source · 1 quote
Gated DeltaNet and Gated Attention core · 1 source · 2 quotes
Gated DeltaNet and Qwen Sparse Attention core · 1 source · 1 quote
hybrid Gated DeltaNet + sparse MoE architecture core · 1 source · 1 quote
Hybrid sparse mixture-of-experts Transformer architecture core · 1 source · 1 quote
Gated Attention and Sliding-Window Attention Configuration used · 1 source · 1 quote
Search-Based Sliding-Window Attention Pattern evaluated · 1 source · 1 quote
Dense Attention Fallback unclear · 1 source · 1 quote

By model

Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modeltechniques
GLM-5.3-Flash coreHybrid Attention core—
NVIDIA-Nemotron-3-Ultra-550B-A55B coreHybrid Mamba-Attention coreHybrid Mamba-Transformer core—
MiMo-V2.6-Flash coreHybrid Attention coreHybrid sparse mixture-of-experts Transformer architecture core—
DeepSeek-V4-Flash usedHybrid Attention used—
MiMo-V2.5 coreHybrid Attention core—
GLM-5.2 evaluatedSearch-Based Sliding-Window Attention Pattern evaluated—
MiniMax-M3 unclearDense Attention Fallback unclear—
DeepSeek-V4-Pro usedHybrid Attention used—
Gemma 4 31B coreHybrid Attention core—
Inkling coreHybrid Attention core—
Kimi K3 coreHybrid Attention core—
Laguna-S-2.1 coreHybrid Attention coreGated Attention and Sliding-Window Attention Configuration used—
MiMo-V2.5-Pro coreHybrid Attention core—
MiMo-V2.6-Pro coreHybrid Attention coreHybrid sparse mixture-of-experts Transformer architecture core—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B coreHybrid Mamba-Transformer coreMamba-2, Mixture-of-Experts, and Selective Attention Hybrid core—
Qwen3.5-397B-A17B coreGated DeltaNet and Full Attention corehybrid Gated DeltaNet + sparse MoE architecture coreHybrid Mamba-Transformer Mixture-of-Experts Layer Layout core—
Qwen3.6-35B-A3B coreGated DeltaNet and Gated Attention core—
Qwen3.8-Flash-Next coreGated DeltaNet and Qwen Sparse Attention coreHybrid Attention core—
gpt-oss-120b usedHybrid Attention used—