Kimi K3
Moonshot AIKimi K3 applies the same construction to the up branch asβ 2 tanh(Wux/β2), preventing either branch from dominating the product.
2 more from this model
we propose Sigmoid Tanh Unit GLU (SiTU-GLU)
Activation Function: SiTU-GLU
specific method · filed under model architecture
SiTU-GLU is a gated feed-forward activation that modifies SwiGLU by applying a smooth tanh-based softcap to the Swish gate's linear factor and to the up branch.1 It controls large-value growth that in SwiGLU can cause activation outliers and overflow risk, while keeping SwiGLU's characteristic near-origin and positive-side response.2,3
Trade-off Unlike SwiGLU, whose response is unbounded, SiTU-GLU bounds large inputs while closely following SwiGLU near the origin.4
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
SiTU-GLU applies the smooth cap softcap(x, β) =βtanh(x/β)to the linear factor of the Swish gate and independently to the up branch— k3 tech report
both multiplicative factors in SwiGLU are unbounded, so coincident large coordinates can produce activation outliers and increase overflow risk in low-precision arithmetic— k3 tech report
controls large-value growth while preserving the characteristic local and positive-side response of SwiGLU— k3 tech report
closely follows SwiGLU near the origin and approaches the bound|f(x)| ≤β1β2 = 100for large positive inputs, whereas SwiGLU remains unbounded— k3 tech report
Also called SiTU-GLU activation function, SiTU-GLU activation (soft-capped SwiGLU), Sigmoid Tanh Unit GLU (SiTU-GLU).
3 quotes from 2 sources, one card per model, each opening with its strongest.
Kimi K3 applies the same construction to the up branch asβ 2 tanh(Wux/β2), preventing either branch from dominating the product.
we propose Sigmoid Tanh Unit GLU (SiTU-GLU)
Activation Function: SiTU-GLU