Open Models Almanac
Methodsmodel architecturetoken mixerhybrid layer stacking

implementation detail · filed under model architecture

Gated DeltaNet and full attention

Combines Gated DeltaNet linear-attention layers with standard or gated attention layers, with some evidence specifying sparse MoE alongside them.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See hybrid layer stacking for the methods filed beside it.

Also called Hybrid Gated DeltaNet and Gated Attention, Hybrid Attention Architecture, Hybrid Attention with QSA.

sources
7
models
4
labs adopt it
2
strongest
core

Documented in

Evidence

9 quotes from 7 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 8

Qwen3.5-397B-A17B

Alibaba (Qwen)adopted3 sources

Gated Delta Networks combined with sparse Mixture-of-Experts deliver high-throughput inference with minimal latency and cost overhead.

coremodel architecturein Qwen3.5-397B-A17BQwen Team
3 more from this model

The model alternates between Gated DeltaNet layers (linear attention) and full attention layers in roughly a 3:1 ratio.

coremodel architecturein Qwen3.5-397B-A17BAlibaba

Hidden Layout: 15 * (3 * (Gated DeltaNet -> MoE) -> 1 * (Gated Attention -> MoE))

coremodel architecturein Qwen3.5-397B-A17BQwen Team

Hidden Layout: 10 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE))

coremodel architecturein Qwen3.5-35B-A3BQwen Team

Qwen3.8-Flash-Next

Alibaba (Qwen)adopted2 sources

Gated DeltaNet and Gated Attention pairing has been reworked into Gated DeltaNet and Qwen Sparse Attention (QSA).

coreunclearin Qwen3.8-Flash-NextQwen Team
1 more from this model

The 3:1 interleave of Gated DeltaNet to full-attention layers is the same ratio Qwen3.5 uses

coremodel architecturein Qwen3.8-Flash-NextQwen Team

Qwen3.6-35B-A3B

Alibaba (Qwen)adopted1 source

It uses a hybrid sparse mixture-of-experts architecture combining Gated DeltaNet linear attention with standard gated attention layers

coremodel architecturein Qwen3.6-35B-A3BAlibaba Cloud
1 more from this model

combining Gated DeltaNet linear attention with standard gated attention layers

coremodel architecturein Qwen3.6-35B-A3BAlibaba Cloud
↑