specific method · filed under model architecture
SiTU-GLU
A GLU activation variant based on the Sigmoid Tanh Unit; the evidence describes a soft-capped or scaled-tanh up branch.
Also called SiTU-GLU activation function, SiTU-GLU activation (soft-capped SwiGLU), Sigmoid Tanh Unit GLU (SiTU-GLU).
- sources
- 2
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 2core 1
Documented in
Evidence
3 spans quoted from the sources, strongest treatment first.
we propose Sigmoid Tanh Unit GLU (SiTU-GLU)
coremodel architecturein Kimi K3Moonshot AI
Activation Function: SiTU-GLU
usedmodel architecturein Kimi K3Moonshot AI
Kimi K3 applies the same construction to the up branch asβ 2 tanh(Wux/β2), preventing either branch from dominating the product.
usedmodel architecturein Kimi K3Moonshot AI
Filed alongside
Other methods under model architecture :: channel mixer :: dense feed-forward network.