Open Models Almanac

Channel mixer · C1

Gated feed-forward units: SwiGLU and its bounded descendants

Multiplying two projections buys quality per parameter and lets a few outputs grow with the square of the weights behind them; a bound gives each unit a known ceiling and pays for it in shape and gradient

Illustration of a stone canal with a wooden sluice gate under a teal hand wheel. A coral stream pours from a copper pipe, swells into a wave as it passes the gate, and runs over a smooth, curved muted-gold lip that flattens it into an even flow along a narrower tiled channel.

Fishman et al. (2024) trained Llama2 7B in FP8, an 8-bit number format, with the scale of each tensor taken from earlier training steps, and watched the run diverge after about 220 billion tokens. The cause was one tensor: the input to the feed-forward unit’s last matrix, where a handful of values had grown far beyond anything seen before. Those values are the output of SwiGLU, the activation that most models in this site’s map use, and the run is the clearest published record of what its product can do.

SwiGLU replaced the single activation function of the original Transformer with the product of two learned projections. On quality per parameter the product wins, and because both of its factors can grow without limit, it can grow like a square. Of the 35 model checkpoints in the map, 32 use a gated feed-forward unit, and by their configs 15 put an explicit bound on it: a hard clamp, or a smooth cap that bends the curve. The limits are small numbers in config.json (swiglu_limit: 7.0, swiglu_limit: 10.0, activation_situ_beta: 4.0). The reports that give a reason name loss spikes and overflow, and a bound that removes them changes the function and the gradient wherever it bites.

Background: the feed-forward unit

Each Transformer layer has a token mixer (attention or one of its relatives), which moves information between positions, and a channel mixer, the feed-forward unit (FFN), which works on each token’s vector on its own. The 00 primer derives its cost: the FFN’s matrices hold most of a dense model’s parameters (64% in LLaMA-7B), and a matrix costs two floating-point operations (FLOPs) per weight per token in the forward pass. In a mixture-of-experts (MoE) model each expert is itself a small FFN; a router picks a few of them per token, and a shared expert, if there is one, runs for every token (C2). Everything here applies to one FFN, whether it is a dense layer, a routed expert or a shared expert.

The original FFN has two matrices and one activation function ϕ\phi (ReLU, later GELU), applied entry by entry:

FFN(x)=Wd ϕ(Wux),Wu∈Rdff×d,  Wd∈Rd×dff,\text{FFN}(x) = W_d\, \phi(W_u x), \qquad W_u \in \mathbb{R}^{d_{\text{ff}} \times d},\; W_d \in \mathbb{R}^{d \times d_{\text{ff}}},

with model width dd and hidden width dffd_{\text{ff}}. We call WuW_u the up projection and WdW_d the down projection. With dff=4dd_{\text{ff}} = 4d the unit has 8d28d^2 parameters and costs 16d216d^2 FLOPs per token. Its input xx is the output of an RMSNorm, so its size is set by the norm’s learned gains, not by the token.

Notation for this article: gg and uu are a hidden unit’s gate and up pre-activations (elsewhere in the series gg counts KV heads; no attention appears here), aa is the unit’s output before the down projection, and τ\tau is a clamp limit.

Gated linear units

A gate made of the input

Dauphin et al. (2016) built a language model out of convolutions and borrowed the gate from the LSTM, which decides how much of each number to pass on by multiplying it with a sigmoid, a value between 0 and 1 computed from the input. Their gated linear unit (GLU) does this inside one layer: one projection of the input is the value, a second projection squashed by a sigmoid is the gate, and the output is their entrywise product (⊙\odot):

GLU(x)=σ(Wgx)⊙Wux.\text{GLU}(x) = \sigma(W_g x) \odot W_u x .

Their argument was about gradients. Write X=WuxX = W_u x and G=WgxG = W_g x. For the LSTM-style unit tanh⁡(X)⊙σ(G)\tanh(X) \odot \sigma(G), which they call the gated tanh unit (GTU), the gradient is tanh⁡′(X)∇X⊙σ(G)+tanh⁡(X)⊙σ′(G)∇G\tanh'(X)\nabla X \odot \sigma(G) + \tanh(X) \odot \sigma'(G)\nabla G; for the GLU it is ∇X⊙σ(G)+X⊙σ′(G)∇G\nabla X \odot \sigma(G) + X \odot \sigma'(G)\nabla G. The GLU’s first path has no factor tanh⁡′(X)\tanh'(X), which is near zero wherever tanh is flat, so gradients reach the value branch scaled only by the gate: “a multiplicative skip connection which helps gradients flow through the layers” (§3). At comparable parameter counts their GLU networks reached lower perplexity than GTU, tanh and ReLU networks.

Shazeer’s variants

Shazeer (2020) put GLUs into the FFN of a Transformer, with a third matrix and a choice of gate function ϕ\phi:

FFNϕGLU(x)=Wd [ ϕ(Wgx)⊙Wux ].\text{FFN}_{\phi\text{GLU}}(x) = W_d\, \big[\, \phi(W_g x) \odot W_u x \,\big].

The sigmoid gives GLU, the identity gives Bilinear, ReLU gives ReGLU, GELU gives GEGLU, and Swish, z σ(βz)z\,\sigma(\beta z), with β=1\beta = 1, silu(z)=z σ(z)\text{silu}(z) = z\,\sigma(z), gives SwiGLU. The primer’s code writes the same unit with W1W_1 (gate), W3W_3 (up) and W2W_2 (down).

To compare at equal cost, Shazeer reduced dffd_{\text{ff}} “by a factor of 2/3”, from 3,072 to 2,048 in a T5-base model with d=768d = 768. From a 4d4d baseline that is dff=83dd_{\text{ff}} = \tfrac{8}{3}d, and 3⋅d⋅83d=8d23 \cdot d \cdot \tfrac{8}{3}d = 8d^2: the same parameters and FLOPs as the two-matrix unit (computed; the primer shows LLaMA’s version of the rule).

Figure 1.The gated unit spends its third matrix by narrowing all three. At d = 4,096, two matrices of width 16,384 and three of width 10,923 both hold 134 million weights (computed).

Table 1 of the paper reports held-out log-perplexity after 65,536 and 524,288 training steps, with all models matched for parameters and computation:

FFN65,536 steps524,288 steps
ReLU (baseline)1.9971.677
GELU1.9831.679
Swish1.9941.683
GLU1.9821.663
Bilinear1.9601.648
ReGLU1.9531.645
GEGLU1.9421.633
SwiGLU1.9441.636

The three ungated activations end within 0.006 of each other. Every gated variant beats all three, Bilinear included, which has no activation at all. (Dauphin’s convolutional model ranked Bilinear well below GLU; the two settings differ in architecture and in parameter matching, so neither result ranks the bilinear form in general.) GEGLU, not SwiGLU, has the lowest number at both step counts, by less than the spread Shazeer measured between repeated short runs of SwiGLU (0.010); his wording is that “the GEGLU and SwiGLU variants produce the best perplexities”. Fine-tuning results mostly favour the gated variants too, with the caveat “the results are noisy”. The paper ends: “We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.”

LLaMA used SwiGLU, and the map shows it stuck. Thirty of the 35 checkpoints pass this site’s swiglu code check, which looks for act(gate(x)) * up(x) with a SiLU-type activation in the modeling code. Gemma 4 uses GEGLU (hidden_activation: "gelu_pytorch_tanh"), as Gemma 1 and 2 said they did. Kimi K3 uses its own gate, SiTU-GLU, described below. The three NVIDIA Nemotron checkpoints drop gating and use a two-matrix FFN with squared ReLU (mlp_hidden_act: "relu2").

What the product computes

Take one hidden unit, with g=wg⊤xg = w_g^\top x and u=wu⊤xu = w_u^\top x, and output a=silu(g) ua = \text{silu}(g)\,u. For large positive gg, σ(g)≈1\sigma(g) \approx 1 and a≈gu=x⊤wgwu⊤xa \approx g u = x^\top w_g w_u^\top x, a quadratic form in the input. For large negative gg the sigmoid closes the gate: SiLU never goes below −0.278-0.278 (at g=−1.28g = -1.28, computed), so only the gate factor’s positive side is unbounded. The output still grows with uu on both sides.

An ungated unit ϕ(w⊤x)\phi(w^\top x) is a ridge function: it changes only along the one direction ww. A gated unit depends on two directions and computes their product exactly; the FFN’s output is then a weighted sum of dffd_{\text{ff}} such products. Multiplicative units of this kind are old (the sigma-pi units of Rumelhart, Hinton and McClelland, 1986), and Jayakumar et al. (2020) show that multiplicative interactions enlarge the set of functions a layer of a given size can represent. Whether that is why gating wins in language models no source we read establishes; the strongest evidence in Shazeer’s table is the Bilinear row, whose only nonlinearity is the product.

Why the product makes outliers

Growth comes from the weights

The input to the FFN is normalized, so what grows during training is the weights. If one unit’s gate and up weights both grow by a factor cc, its output on a fixed input grows by c2c^2 where the gate is open, while a ReLU unit’s grows by cc. Along g=u=sg = u = s, SwiGLU gives 0.73 at s=1s = 1, 15.7 at s=4s = 4, 64.0 at s=8s = 8 and 1,024 at s=32s = 32 (computed); GELU gives about ss.

How large the product can get depends on how the two weight vectors point. For any vectors pp and qq, x⊤pq⊤x=x⊤Mxx^\top p q^\top x = x^\top M x with M=12(pq⊤+qp⊤)M = \tfrac12(pq^\top + qp^\top), the symmetric part (the antisymmetric part contributes nothing to x⊤⋅ xx^\top \cdot\, x). MM has rank two, with eigenvectors p/∥p∥±q/∥q∥p/\|p\| \pm q/\|q\| and eigenvalues 12(p⊤q±∥p∥∥q∥)\tfrac12(p^\top q \pm \|p\|\|q\|). So over inputs of a fixed norm,

∣g u∣≤12 ∥wg∥ ∥wu∥ (1+∣cos⁡θ∣) ∥x∥2,|g\,u| \le \tfrac12\, \|w_g\|\,\|w_u\|\,\big(1 + |\cos\theta|\big)\, \|x\|^2,

where θ\theta is the angle between wgw_g and wuw_u, and the worst input points halfway between them. After an RMSNorm with gains γ\gamma, ∥x∥2≤dmax⁡jγj2\|x\|^2 \le d \max_j \gamma_j^2. Aligned weights (or anti-aligned ones, cos⁡θ=−1\cos\theta = -1) double the ceiling of orthogonal weights, and they make large gate values and large up values arrive on the same inputs.

Fishman et al. argue that training produces this alignment. Their Theorem 1 considers a stationary point of the loss with an ℓ2\ell_2 penalty on the weights (a point where the gradient of loss plus penalty is zero) at which one unit’s gate sigmoid is saturated, flat at 1 on every training token. There, the gate and up weights satisfy wu=Awgw_u = A w_g and wg=Awuw_g = A w_u for one symmetric matrix AA, which they read as wg=±wuw_g = \pm w_u. (The equations force equal norms; exact alignment needs one more condition on AA.) The penalty picks the direction; weak regularization lets the norms grow, which they say strengthens the effect. In their Llama2 7B run, trained with AdamW, the correlation between the two weight vectors of an outlier channel “is initially low, but then it increases drastically between 125B and 210B tokens”, together with further norm growth, and the outliers appeared after about 200 billion tokens. They saw strongly negative correlations in other channels too.

|silu(g)·u|, relative to the largesttop 1%

value at s = 1 (shown once s > 1)

GELU since s = 1
×1.0
SwiGLU since s = 1
×1.0
largest ÷ median |silu(g)·u|
71×

2,000 synthetic tokens: p, q standard normal (seeded), g = s·p, u = s·(ρp + √(1−ρ²)·q). GELU in its tanh form.

Figure 2.Synthetic pre-activations, not a trained model; s stands for growth in the weights, since the FFN's input is normalized. The largest output of a one-branch activation grows in proportion to s, the largest SwiGLU output with s². In this sample, raising ρ from 0 to 0.9 doubles the largest SwiGLU output. In trained models the large values sit in a few channels and tokens, not in a uniformly wider cloud.

Where the large values land

The product a=silu(g)⊙ua = \text{silu}(g) \odot u is the input to the down projection, and that is where Fishman et al. found the trouble: “the SwiGLU activation function can lead to outliers in the input of the last linear layer of the MLP component.” Keeping only that tensor in higher precision made their FP8 run converge. Yang et al. (2024) found the same place from the inference side: quantizing LLaMA-2, LLaMA-3, Mistral, Mixtral and Gemma activations to 8-bit integers after training, they saw “activation spikes” at the input of the FFN’s down projection, in “the early and late layers”, and “dedicated to a couple of tokens”.

Yu et al. (2024) traced the residual stream’s “massive activations” (Sun et al., 2024: a few entries thousands of times larger than the rest) back to this product, amplified by a single large weight, the “super weight”, in an early layer’s down projection.

What a large value does to a low-precision number

FP8 E4M3 stores a number in eight bits: a sign, four exponent bits and three mantissa bits. Its largest finite value is 448. Its normal numbers run from 2−62^{-6} to 448, a ratio of about 28,700, with eight steps between one power of two and the next, so rounding changes a value by at most 1/16 of itself anywhere in that range; below 2−62^{-6}, precision thins out until values round to zero (computed from the bit layout). To cast a tensor to FP8, it is multiplied by a scale that puts its largest values near 448.

Fishman et al. used delayed scaling: one scale per tensor, taken from the largest values of earlier steps. Their failure was the delay. A value that has grown since the scale was set lands above 448 and is clipped, which they describe as a spike that “disrupts” the continuity delayed scaling assumes. That is the setting where one outlier does the most damage.

The labs in the map scale at a finer grain. DeepSeek-V4 trains with FP8 in blocks of 128×128 for weights and casts the SwiGLU output to FP8 right before the second expert matrix multiplication (its Fig. 5 shows “SwiGLU + FP8 Cast” between the two GEMMs); Kimi K3 stores most activations with block-wise FP8. With a scale per block of 32 to 128 values computed when the block is cast, an outlier affects only its own block, in one token’s row. Within that block, E4M3 still keeps every value larger than 1/28,700 of the outlier at full relative precision, so a block-scaled FP8 cast tolerates outliers well.

FP4 E2M1 has the nonzero magnitudes 0.5, 1, 1.5, 2, 3, 4 and 6, a ratio of 12. In a block scaled so its largest value lands at 6, every value less than 1/24 of that largest one rounds to zero (computed). The MX formats (MXFP4, MXFP8) share one power-of-two scale per block of 32 values. 8-bit integers, Yang et al.’s setting, have a uniform grid and lose small values the same way.

31 typical values, 0.05 to 1outlierto the value it rounds to

scale
×224.0
values clipped
1
below E4M3's normal range
0
rounded to zero
0
worst relative error, typical values
5.9%

Delayed scale 448 / 2: set when the block's largest value was 2. Current scales 448 / outlier (FP8) and 6 / outlier (FP4), not rounded to powers of two. Round to nearest, ties to even.

Figure 3.One block of 32 values with one outlier, idealized (scales not rounded to powers of two). With a per-tensor scale set before the outlier grew, the outlier is clipped at 448. With a scale computed for this block, FP8 represents every value to within 1/16. In FP4 the same block loses every value smaller than 1/24 of the outlier.

Smooth-SwiGLU (Fishman et al.) is a response at this level that leaves the function alone. It multiplies channel ii of the up branch by a factor sis_i before the FP8 cast and folds 1/si1/s_i into column ii of the down projection, so Wda=(WdS−1)(Sa)W_d a = (W_d S^{-1})(S a) and the network computes the same thing at “zero additional cost at inference”. On Llama2 7B it reached results “on par” with BF16.

The bounds the models in the map ship change the function instead. The reports that give a reason name stability. DeepSeek-V4: “the occurrence of spikes is consistently tied to outliers in the MoE layers, and the routing mechanism itself appears to exacerbate the emergence of these outliers” (§4.2.3). Kimi K3: at extreme expert sparsity its routed path is “a chain of nearly four consecutive matrix multiplications” that “produces exploding internal activations”, and “both multiplicative factors in SwiGLU are unbounded, so coincident large coordinates can produce activation outliers and increase overflow risk in low-precision arithmetic” (§2.3). Neither ties its bound to a specific format.

Clamped SwiGLU

gpt-oss: clamp, sharpen, and add one

The gpt-oss model card says only this, in a footnote to §2.2: “Our SwiGLU implementation is unconventional, including clamping and a residual connection.” The modeling code (GptOssExperts._apply_gate) says the rest. Gate and up are stored interleaved in one matrix, gate in the even columns and up in the odd ones:

gate, up = gate_up[..., ::2], gate_up[..., 1::2]
gate = gate.clamp(min=None, max=self.limit)
up = up.clamp(min=-self.limit, max=self.limit)
glu = gate * torch.sigmoid(gate * self.alpha)
gated_output = (up + 1) * glu

with swiglu_limit: 7.0 in the config and α=1.702\alpha = 1.702 from the config class’s default. With τ=7\tau = 7:

a=(clip(u,−τ,τ)+1)⋅g^,g^=g′ σ(1.702 g′),g′=min⁡(g,τ).a = \big(\text{clip}(u, -\tau, \tau) + 1\big) \cdot \hat g, \qquad \hat g = g'\,\sigma(1.702\, g'),\quad g' = \min(g, \tau).

The gate is clamped from above only, since the sigmoid already bounds it below; the up branch is clamped on both sides, because it enters linearly. The gate function z σ(1.702z)z\,\sigma(1.702 z) is the standard sigmoid approximation of GELU, so this gate is closer to GEGLU’s than to SwiGLU’s (our reading; the code doesn’t say). And the +1+1 splits the output into a=clip(u) g^+g^a = \text{clip}(u)\,\hat g + \hat g: a gated unit plus an ungated GELU-like unit that shares the gate’s weights. That is presumably the footnote’s residual connection.

The output lies in [−42,56][-42, 56] (computed: (±7+1)⋅7 σ(11.9)(\pm 7 + 1) \cdot 7\,\sigma(11.9)). The +1+1 and the sharper gate also change the unit below the clamp: at g=u=1g = u = 1 it outputs 1.69 where SwiGLU outputs 0.73, 2.3 times as much (computed). The code check, which looks for a SiLU-type gate times an up projection, counts it as SwiGLU.

MiniMax-M3 and M3.1-Flash run the same formula, with a code comment that says so (“same as GPT OSS, but the weights are not interleaved”) and both values in the config (swiglu_alpha: 1.702, swiglu_limit: 7.0), and apply it to routed experts, the shared expert and the three dense layers; their documents never mention it.

DeepSeek-V4 and the plain clamp

DeepSeek-V4’s report states its clamp’s limits and its reason: “we clamped the linear component of SwiGLU to the range of [−10,10][-10, 10], while capping the upper bound of the gate component at 10”, which “effectively eliminates outliers and substantially aids in stabilizing the training process, without compromising performance” (§4.2.3). The DeepSeek-V4.1 report repeats “clamping … at a threshold of 10” without a reason. The report cites gpt-oss for the idea, but the code keeps SwiGLU itself, with no α\alpha and no +1+1:

a=silu(min⁡(g,τ))⋅clip(u,−τ,τ),τ=10.a = \text{silu}\big(\min(g, \tau)\big) \cdot \text{clip}(u, -\tau, \tau), \qquad \tau = 10 .

This is SwiGLU evaluated at the nearest point of the box g≤τg \le \tau, ∣u∣≤τ|u| \le \tau: unchanged inside it, and outside it the clipped coordinate stops mattering. The output lies within ± silu(10)⋅10=±99.995\pm\,\text{silu}(10) \cdot 10 = \pm 99.995 (computed). Hy4-preview’s and GLM-5.3-Flash’s modeling code implement the same clamped unit, each with the comment “Simple swiglu instead of alpha”.

The cost shows in the gradient. Where g>τg > \tau, the output no longer depends on gg, so no gradient reaches the gate weights through that unit for that token; where ∣u∣>τ|u| > \tau, the same holds for the up weights. When both are clipped, the unit sends nothing back to its input or to its gate and up weights, though its column of the down projection still trains. A straight-through estimator would pass the gradient as if the clamp were not there; these models don’t. (This is activation clipping, unrelated to gradient clipping, which rescales the whole gradient.) A unit clipped on many tokens stops learning from them, as a dead ReLU does.

Nothing trains the limit itself: τ\tau is a constant in the config. Weight decay supplies a restoring force, though. A clipped token sends no gradient to the clipped weights, but decay keeps shrinking them, so a unit clipped on every token is pulled back towards the box, and the saturated stationary point of Fishman’s theorem is not available to it. That is our inference; whether it is why the clamp works, the reports don’t test. DeepSeek’s report lists the clamp among techniques whose “underlying principles remain insufficiently understood” (§6).

Step: the same clamp with smaller limits

StepFun’s checkpoints clamp after SiLU rather than before it, a=min⁡(silu(g),τ)⋅clip(u,−τ,τ)a = \min(\text{silu}(g), \tau) \cdot \text{clip}(u, -\tau, \tau). Since silu(τ)=τ(1−σ(−τ))\text{silu}(\tau) = \tau(1 - \sigma(-\tau)), the two placements change the gate factor by at most τ σ(−τ)=0.006\tau\,\sigma(-\tau) = 0.006 at τ=7\tau = 7, and the output by at most 0.045 (computed), so Step’s unit is the plain clamp with a smaller limit. What differs is where it applies. In Step-3.7-Flash, swiglu_limits is 7 for the routed experts of layers 43 and 44 (counting from 0, the last two of 45) and 0, meaning no clamp, elsewhere; swiglu_limits_shared is 16 for the shared expert of the same two layers. The output ceilings are 7⋅7=497 \cdot 7 = 49 and 16⋅16=25616 \cdot 16 = 256 (computed). The documents don’t say why those two layers were chosen; Yang et al.’s spikes concentrate in early and late layers, but linking the two is our guess. In the next release, Step-5-preview, both lists are 7.0 for every layer, so every FFN is clamped. (Its kernel is in a vLLM file, vLLM being a common serving library, that the code check didn’t fetch; that it clamps after the activation like Step-3.7 is our inference from its name, SwigluStepAndMul.)

  • SwiGLU silu(g)·u
  • plain clamp τ = 10: silu(min(g, τ))·clip(u, ±τ) (DeepSeek-V4, Hy4, GLM-5.3-Flash)
  • plain clamp τ = 7 (Step's routed experts, within 0.045)
  • gpt-oss τ = 7: (clip(u, ±7) + 1)·g′σ(1.702 g′), g′ = min(g, 7) (gpt-oss, MiniMax)

(g, u) plane, −20 to 20: plain τ = 10

a > 0a < 0clamps inactive

Plain clamp (DeepSeek-V4, Hy4, GLM-5.3-Flash), τ = 10: equals SwiGLU inside g ≤ 10, |u| ≤ 10. Output range −100.0 to 100.0; largest |a| = 99.995.

Figure 4.Along the line where gate and up inputs are equal, a plain clamp follows SwiGLU exactly until the limit and then stays flat: at 100 for τ = 10, at 49 for τ = 7. gpt-oss's unit flattens at 56 but differs from SwiGLU everywhere; zoomed in, it gives 1.69 at s = 1 where SwiGLU gives 0.73.

Who clamps what

A config field doesn’t say which FFNs it covers; that is visible only in the modeling code, and the labs differ:

CheckpointsUnitLimitFFNs clamped (from the code)
gpt-oss-120bgpt-oss7all (it has only routed experts)
MiniMax-M3, M3.1-Flashgpt-oss7routed, shared, dense
DeepSeek-V4-Flash, V4-Pro (5 checkpoints)plain clamp10routed, shared (no dense layers)
DeepSeek-V4.1-Flash (2 checkpoints)plain clamp10config and report only; not visible in the fetched code
Hy4-previewplain clamp10routed only; shared expert and dense layer unclamped
GLM-5.3-Flashplain clamp10routed, shared, dense, vision tower
Step-3.7-Flashplain clamp, after SiLU7 / 16routed / shared, layers 43–44 only
Step-5-previewplain clamp, after SiLU (inferred)7all layers

The choice moves between releases of one lab: DeepSeek-V3.2 and Tencent’s Hy3 have no bound, DeepSeek-V4 and Hy4-preview do, and Z.ai bounds GLM-5.3-Flash but not GLM-5.3.

A clamp is part of the model

A checkpoint trained with a clamp has to be served with it. Where the clamp binds, removing it feeds the down projection values it never saw in training, and the outputs change; how often it binds, no report gives. The configs make it easy to get wrong. gpt-oss’s says "hidden_act": "silu", which its expert code ignores, so a loader that trusts the field computes plain SwiGLU and drops the clamp, the α\alpha and the +1+1. MiniMax’s says "swigluoai", the name of a serving kernel for the gpt-oss unit, which the config class overrides. DeepSeek-V4’s code puts the clamp in _apply_gate “so the grouped_mm / batched_mm backends … apply the same clamp + SiLU on top of their packed gate_up output instead of bypassing it”. Step-5’s serving code refuses any other value (“Step5 fused MoE supports only swiglu_limit=7.0”). The layout is part of it too: an interleaved gate and up (gpt-oss) loaded as two halves (MiniMax) pairs the wrong columns.

SiTU-GLU: a smooth cap

Kimi K3 bounds the unit without a clamp. Its Sigmoid-Tanh Unit GLU (SiTU-GLU) applies a scaled tanh, softcap(z,β)=βtanh⁡(z/β)\text{softcap}(z, \beta) = \beta \tanh(z/\beta), to the linear factor of the gate and to the up branch:

a=β1tanh⁡ ⁣(g/β1) σ(g)⋅β2tanh⁡ ⁣(u/β2),β1=4, β2=25.a = \beta_1 \tanh\!\big(g/\beta_1\big)\, \sigma(g) \cdot \beta_2 \tanh\!\big(u/\beta_2\big), \qquad \beta_1 = 4,\ \beta_2 = 25 .

The config fields are activation_situ_beta: 4.0 for the gate and activation_situ_linear_beta: 25.0 for the up branch, and the code computes the unit in float32. K3 uses it in every FFN: the dense first layer, the two shared experts and the 896 routed experts. The same tanh cap was used on attention logits before (Gemma 2 caps logits with 50tanh⁡(⋅/50)50\tanh(\cdot/50)), and SiTU’s up branch is Dauphin’s GTU tanh with a temperature of 25: Dauphin et al. removed that tanh to keep gradients flowing, and Kimi puts it back, scaled so it is nearly linear where most values are and saturates only where the bound is meant to act.

Since ∣tanh⁡∣<1|\tanh| < 1 and 0<σ<10 < \sigma < 1, every output satisfies ∣a∣<β1β2=100|a| < \beta_1 \beta_2 = 100 (Appendix B). The cap bounds the gate factor’s positive side (its negative side stays closed by the sigmoid) and both sides of the up factor. The bound is approached slowly; at g=u=60g = u = 60 the output is 98.4 (computed).

Near zero, βtanh⁡(z/β)=z (1−z2/3β2+… )\beta \tanh(z/\beta) = z\,(1 - z^2/3\beta^2 + \dots), so each factor is off by a term of third order, and the relative gap to SwiGLU is about −g2/48−u2/1875-g^2/48 - u^2/1875. Along g=u=sg = u = s that gives the computed gaps: 0.5% at s=0.5s = 0.5, 2.1% at s=1s = 1, 7.8% at s=2s = 2, then 24% at s=4s = 4 and 53% at s=8s = 8, almost all from the gate’s β1=4\beta_1 = 4; the up factor with β2=25\beta_2 = 25 is 0.21% below linear at s=2s = 2. The report’s claim that SiTU-GLU “matches SwiGLU to first order around the origin” is this expansion. As both β\beta go to infinity it becomes SwiGLU at every point, but for finite β\beta the two separate without limit.

The report’s case against hard clamping is the gradient: “Unlike hard clamping of gate pre-activations, the smooth cap preserves nonzero gradients away from saturation boundaries, which we find to give better training behavior” (Appendix B). The gradient is positive everywhere but small over much of the range. The derivative of βtanh⁡(z/β)\beta\tanh(z/\beta) is sech2(z/β)\text{sech}^2(z/\beta), which decays like 4e−2z/β4e^{-2z/\beta}. With β1=4\beta_1 = 4 the gate factor’s slope is 0.47 at g=4g = 4, 0.07 at g=8g = 8 and 0.03 at g=10g = 10 (computed), where a τ=10\tau = 10 clamp still passes SiLU’s full slope of about 1. The up factor’s slope is 0.86 at u=10u = 10 and 0.0013 at u=100u = 100. A clamp passes the whole gradient up to its limit and none beyond; the smooth cap weakens it across the range and leaves a small tail, the old trade-off between hard-tanh and tanh. The report gives no ablation numbers for SiTU-GLU against SwiGLU or against a clamp; its scaling-efficiency gain over Kimi K2 is for all of K3’s changes together.

change β₁ and β₂
  • SwiGLU
  • SiTU-GLU β₁·tanh(g/β₁)·σ(g) · β₂·tanh(u/β₂)
  • plain clamp τ = 10
at s =
1.0
SwiGLU
0.731
SiTU-GLU
0.716
clamp τ = 10
0.731
SiTU vs SwiGLU
−2.1%
gate slope at g = s: SiLU
0.928
clamp τ = 10
0.928
SiTU gate
0.880
Figure 5.Top: along the line where gate and up inputs are equal, Kimi K3's SiTU-GLU stays within about 2% of SwiGLU up to s = 1 and bends towards its bound of 100 instead of stopping at a corner. Bottom: the gate factor's slope. A clamp at 10 passes SiLU's full slope up to 10 and none beyond; SiTU's slope falls from 0.47 at g = 4 to 0.03 at g = 10 and never reaches zero.

The cap’s other cost is arithmetic per element, which counts only where the matrix units are busy. SiLU needs an exponential and a division per hidden unit; SiTU-GLU adds two tanh evaluations. These run on the GPU’s special-function units, which have a small fraction of the throughput of the units that do the matrix multiplications. In decode the FFN is limited by reading weights, and the difference disappears. In training and prefill, where the matrix multiplications are compute-bound and run in FP8 or FP4, the activation is computed between the two matrix multiplications and can hold them up: DeepSeek-V4 proposes, as a suggestion to hardware designers, “replacing SwiGLU with a low-cost element-wise activation that involves no exponential or division operations” (§3.1). The clamps cost two comparisons.

Kimi K3 lists an RMSNorm among the same stabilizers. It sits outside the expert. K3’s experts run in a narrower latent space (3,584 instead of 7,168), and a shared projection maps their summed output back to full width; the RMSNorm normalizes that sum before the projection. C2 covers this latent MoE.

What a ceiling bounds

A bound holds for one hidden unit before the down projection. If every ∣ai∣≤C|a_i| \le C, the FFN’s output satisfies ∥Wda∥∞≤Cmax⁡j∑i∣Wd,ji∣\|W_d a\|_\infty \le C \max_j \sum_i |W_{d,ji}|: finite, but still free to grow through the down projection’s weights, which is the route Yu et al.’s super weight takes. An MoE layer adds its routing weights; DeepSeek-V4’s routed contributions sum with weight 1.5 (norm_topk_prob: true, routed_scaling_factor: 1.5). So a clamp removes two of the three factors that can grow (wgw_g and wuw_u) and leaves the third.

What a known ceiling buys a quantizer is, in our reading, a scale that needs no history: with ∣a∣≤100|a| \le 100, a fixed FP8 scale of 448/100448/100 can never overflow and still represents values down to about 0.0035 at full precision (computed). Under fresh per-block scaling that matters less, which fits the reports citing stability rather than a format. A pre-activation clamp also bounds gg and uu themselves, which some training stacks keep in FP8 for recomputation; SiTU-GLU bounds only aa.

Comparison

SwiGLUgpt-oss unitPlain clamp (DeepSeek, Hy4, GLM, Step)SiTU-GLU
Output aasilu(g) u\text{silu}(g)\,u(clip(u)+1) g′σ(1.702g′)(\text{clip}(u)+1)\,g'\sigma(1.702g')silu(min⁡(g,τ)) clip(u)\text{silu}(\min(g,\tau))\,\text{clip}(u)4tanh⁡(g/4)σ(g)⋅25tanh⁡(u/25)4\tanh(g/4)\sigma(g)\cdot 25\tanh(u/25)
Range of aaunbounded[−42,56][-42, 56] (τ=7\tau = 7)±49\pm 49 (τ=7\tau = 7), ±100\pm 100 (τ=10\tau = 10), ±256\pm 256 (τ=16\tau = 16)(−100,100)(-100, 100)
Equal to SwiGLUeverywhereonly at g=0g = 0inside the boxto first order near 0
Gate gradientSiLU’sfull up to τ\tau, zero beyondfull up to τ\tau, zero beyondreduced from g≈2g \approx 2 (0.47 at 4, 0.03 at 10), never zero
Extra elementwise worknonetwo clamps, a +1+1two clampstwo tanh
Reason given in a reportquality (Shazeer)noneMoE outliers and loss spikes (DeepSeek-V4)activation explosion, overflow risk (K3)
Checkpoints in the map16311 (2 config and report only, 1 inferred kernel)1

Two fixes leave the unit alone. Smooth-SwiGLU rescales around the quantizer, as above. And Qwen3.8-Flash-Next, a plain SwiGLU model, trained “without a single loss spike”, “without relying on explicit clipping methods such as qk-clip … or SwiGLU-clip”, crediting the sigmoid gates it puts on what attention, Gated DeltaNet and its Gated Residual write into the residual stream (T6, R2). The same gates let it store the residual state in FP8 at inference.

Where the models in the map sit

The 35 checkpoints split six ways on their FFN unit (from the modeling code, except DeepSeek-V4.1, whose two clamped checkpoints rest on config and report; counts are checkpoints, so DeepSeek’s five V4 checkpoints count five times):

FFN unitCheckpointsLabs
SwiGLU, unbounded16Aleph Alpha, DeepSeek (V3.2), Liquid, Mistral, Poolside, Qwen (3), Tencent (Hy3), Thinking Machines, Xiaomi (4), Z.ai (GLM-5.2, 5.3)
SwiGLU, plain clamp11DeepSeek (V4 ×5, V4.1 ×2), Tencent (Hy4), Z.ai (GLM-5.3-Flash), StepFun (2)
gpt-oss unit, clamped3OpenAI, MiniMax (2)
SiTU-GLU1Moonshot (Kimi K3)
GEGLU, unbounded1Google (Gemma 4)
two-matrix ReLU², no gate3NVIDIA

Seven of the 16 labs bound their unit in at least one checkpoint. The swiglu code check counts 30 present, because it reads the clamped units and the gpt-oss unit as SwiGLU and SiTU-GLU as something else; the table splits that count by what the code computes.

The models’ FFN widths do not follow Shazeer’s ⅔ rule. Dense FFN layers (most MoE models keep one to three at the bottom) range from 2.0d2.0d (GLM-5.2, GLM-5.3, MiniMax-M3 and M3.1-Flash) to 4.71d4.71d (Kimi K3’s first layer, 33,792 at d=7,168d = 7{,}168). Two checkpoints land exactly on 83d\tfrac{8}{3}d: MiMo-V2.5-Pro and V2.6-Pro, with 16,384 at d=6,144d = 6{,}144. Six keep three matrices at the full 4d4d, 50% more weights than the matched width: Gemma 4, Mistral Large 4, Laguna S 2.1, Inkling, MiMo-V2.5 and MiMo-V2.6-Flash. Experts are much narrower, from 0.2d0.2d (Kolibri-1) to 1.0d1.0d (gpt-oss-120b, Mistral Large 4).

Most bounded models ship low-precision weights (MXFP4 in gpt-oss, FP4 experts in DeepSeek-V4, FP8 in GLM-5.3-Flash and Step-5-preview), but that is storage, often added in post-training: DeepSeek-V4’s FP4 comes from quantization-aware training after pre-training, while its clamp ran “throughout the training”.

Easy to get wrong

  • “SwiGLU won Shazeer’s comparison.” Gating won; GEGLU and SwiGLU tie within noise (Table 1).
  • “A clamped SwiGLU is SwiGLU with a ceiling.” True of the plain clamp; the gpt-oss and MiniMax unit differs from SwiGLU everywhere (see “gpt-oss: clamp, sharpen, and add one”).
  • “swiglu_limit clamps the whole FFN.” Only what the code applies it to; see “Who clamps what”.
  • “hidden_act tells you the activation.” For gpt-oss, MiniMax and Step it doesn’t; a loader that trusts it drops the clamp.
  • “Outliers come from large inputs.” The FFN’s input is normalized; the growth is in the weights and in how the gate and up weights align.
  • “The bound protects FP8.” The reports name stability and overflow in general, and block-scaled FP8 tolerates single outliers well; four-bit formats and stale per-tensor scales are where one value does real harm.
  • “A smooth cap passes more gradient than a clamp.” Only beyond the clamp’s limit; inside it the clamp passes SiLU’s full slope and the cap much less.

Check yourself

  1. A model has d=4,096d = 4{,}096. What hidden width gives a SwiGLU FFN the same parameter count as a two-matrix FFN of width 4d4d, and how many parameters is that?

    Answerdff=23⋅4d=83⋅4,096≈10,923d_{\text{ff}} = \tfrac{2}{3} \cdot 4d = \tfrac{8}{3} \cdot 4{,}096 \approx 10{,}923. Both have 8d2=134,217,7288d^2 = 134{,}217{,}728 parameters, about 134 million.

  2. Compute the range of gpt-oss’s unit with τ=7\tau = 7, and say at which gg and uu its ends are reached.

    AnswerThe gate is capped at g′=7g' = 7, giving g^=7 σ(1.702⋅7)=7 σ(11.9)≈7.0\hat g = 7\,\sigma(1.702 \cdot 7) = 7\,\sigma(11.9) \approx 7.0. The up factor clip(u)+1\text{clip}(u) + 1 runs from −6-6 to 88. So aa runs from about −42-42 (for g≥7g \ge 7, u≤−7u \le -7) to about 56 (for g≥7g \ge 7, u≥7u \ge 7).

  3. In DeepSeek-V4’s unit, a token has g=14g = 14 and u=3u = 3 at one hidden unit. Which weights get a gradient through this unit for this token?

    AnswerThe gate is clipped (14>1014 > 10), so a=silu(10)⋅3a = \text{silu}(10) \cdot 3 doesn’t depend on gg and the gate weights get nothing. u=3u = 3 is inside the box, so ∂a/∂u=silu(10)≈10\partial a / \partial u = \text{silu}(10) \approx 10 and the up weights get a gradient. The unit’s column of the down projection gets one too, since a≠0a \ne 0.

  4. Estimate SiTU-GLU’s relative gap to SwiGLU at g=u=1g = u = 1 without a calculator, and say which of the two caps causes most of it.

    AnswerAbout −1/48−1/1875≈−2.1%-1/48 - 1/1875 \approx -2.1\%. The gate’s cap, β1=4\beta_1 = 4, gives −1/48≈−2.08%-1/48 \approx -2.08\%; the up branch’s β2=25\beta_2 = 25 only −0.05%-0.05\%.

  5. A block of 32 values has one entry of 60 and the rest near 1. Scaled so that 60 lands at the format’s maximum, what happens to the 1s in FP8 E4M3 and in FP4 E2M1?

    AnswerE4M3: the scale is 448/60≈7.5448/60 \approx 7.5, so the 1s become about 7.5, well inside the normal range, with at most 1/16 rounding error. E2M1: the scale is 6/60=0.16/60 = 0.1, so the 1s become 0.1, below half the smallest nonzero value (0.5), and round to zero.

In the map

Methods

Taxonomy nodes

Further reading

  • Dauphin et al., Language Modeling with Gated Convolutional Networks (2016), arXiv:1612.08083. The GLU, its gradient argument against the GTU, and the learning curves in a convolutional language model.
  • Shazeer, GLU Variants Improve Transformer (2020), arXiv:2002.05202. Five pages; the variants, the parameter matching, and the fine-tuning tables summarized here.
  • Fishman et al., Scaling FP8 training to trillion-token LLMs (2024), arXiv:2409.12517. The alignment theorem and its assumptions, the late-training outliers, and Smooth-SwiGLU.
  • Yang et al., Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs (2024), arXiv:2405.14428. Where spikes sit in trained models, and two inference-time fixes.
  • Yu et al., The Super Weight in Large Language Models (2024), arXiv:2411.07191. How one weight in a down projection turns the GLU product into the residual stream’s largest activations.
  • Jayakumar et al., Multiplicative Interactions and Where to Find Them (ICLR 2020). What multiplicative units add to a layer’s expressivity, with the classical lineage.
  • OpenAI, gpt-oss-120b & gpt-oss-20b Model Card (2025), arXiv:2508.10925, §2.2. The one-line footnote; read it with the modeling code.
  • DeepSeek-AI, DeepSeek-V4 technical report (2026), arXiv:2606.19348, §4.2.3, §3.1 and Fig. 5. The clamp, the loss-spike motivation, where the FP8 cast sits, and the case for cheaper activations. Use the PDF; the HTML loses the numbers.
  • Kimi Team, Kimi K3: Open Frontier Intelligence (2026), §2.3 and Appendix B. SiTU-GLU’s definition, Figure 4, and the bound and first-order-match argument.
↑