Open Models Almanac
Methodsmodel architecturechannel mixerdense feed-forward network

specific method · filed under model architecture

SiTU-GLU

SiTU-GLU is a gated feed-forward activation that modifies SwiGLU by applying a smooth tanh-based softcap to the Swish gate's linear factor and to the up branch.1 It controls large-value growth that in SwiGLU can cause activation outliers and overflow risk, while keeping SwiGLU's characteristic near-origin and positive-side response.2,3

Trade-off Unlike SwiGLU, whose response is unbounded, SiTU-GLU bounds large inputs while closely following SwiGLU near the origin.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. SiTU-GLU applies the smooth cap softcap(x, β) =βtanh(x/β)to the linear factor of the Swish gate and independently to the up branch — k3 tech report
  2. both multiplicative factors in SwiGLU are unbounded, so coincident large coordinates can produce activation outliers and increase overflow risk in low-precision arithmetic — k3 tech report
  3. controls large-value growth while preserving the characteristic local and positive-side response of SwiGLU — k3 tech report
  4. closely follows SwiGLU near the origin and approaches the bound|f(x)| ≤β1β2 = 100for large positive inputs, whereas SwiGLU remains unbounded — k3 tech report

Also called SiTU-GLU activation function, SiTU-GLU activation (soft-capped SwiGLU), Sigmoid Tanh Unit GLU (SiTU-GLU).

sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Moonshot AI

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 1

Kimi K3

Moonshot AIadopted2 sources

Kimi K3 applies the same construction to the up branch asβ 2 tanh(Wux/β2), preventing either branch from dominating the product.

usedmodel architecturein Kimi K3Moonshot AI
2 more from this model

we propose Sigmoid Tanh Unit GLU (SiTU-GLU)

coremodel architecturein Kimi K3Moonshot AI

Activation Function: SiTU-GLU

usedmodel architecturein Kimi K3Moonshot AI
↑