implementation detail · filed under model architecture
Gated DeltaNet with bounded sigmoid output gate
A Gated DeltaNet variant using a bounded sigmoid output gate instead of the original SiLU output gate.
Also called Sigmoid output gate for Gated DeltaNet, bounded sigmoid gate.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
Unlike the original GDN, which uses a SiLU output gate, we use the bounded sigmoid gate
usedunclearin Qwen3.8-Flash-NextQwen
Filed alongside
Other methods under model architecture :: token mixer :: linear attention & state space :: gated delta network.
Gated DeltaNetKimi Delta AttentionGated DeltaNet–sparse MoE hybridKDAGated DeltaNet with reduced KV-head configurationHybrid linear attentionKimi Delta Attention and Attention Residuals architectureKimi Delta Attention with input-dependent full-rank output gateKimi Delta Attention with lower-bounded log-decaySimpleGDN