implementation detail · filed under model architecture
Hybrid Mamba-Transformer Mixture-of-Experts Layer Layout
Describes a hybrid mixture-of-experts layout that interleaves Gated DeltaNet and gated-attention blocks, each paired with MoE.
- sources
- 2
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
core 2
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
Hidden Layout: 15 * (3 * (Gated DeltaNet -> MoE) -> 1 * (Gated Attention -> MoE))
coremodel architecturein Qwen3.5-397B-A17BQwen
Hidden Layout: 10 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE))
coremodel architecturein Qwen3.5-35B-A3BQwen
Filed alongside
Other methods under model architecture :: token mixer :: hybrid layer stacking.
Hybrid AttentionMamba-2, Mixture-of-Experts, and Selective Attention HybridHybrid Mamba-TransformerHybrid Mamba-AttentionGated DeltaNet and Gated AttentionDense Attention FallbackGated Attention and Sliding-Window Attention ConfigurationGated DeltaNet and Full AttentionGated DeltaNet and Qwen Sparse Attentionhybrid Gated DeltaNet + sparse MoE architectureHybrid sparse mixture-of-experts Transformer architectureSearch-Based Sliding-Window Attention Pattern