Fishman et al. (2024) trained Llama2 7B in FP8, an 8-bit number format, with the scale of each tensor taken from earlier training steps, and watched the run diverge after about 220 billion tokens. The cause was one tensor: the input to the feed-forward unit’s last matrix, where a handful of values had grown far beyond anything seen before. Those values are the output of SwiGLU, the activation that most models in this site’s map use, and the run is the clearest published record of what its product can do.
SwiGLU replaced the single activation function of the original Transformer with the product of
two learned projections. On quality per parameter the product wins, and because both of its
factors can grow without limit, it can grow like a square. Of the 35 model checkpoints in the
map, 32 use a gated feed-forward unit, and by their configs 15 put an explicit bound on it: a
hard clamp, or a smooth cap that bends the curve. The limits are small numbers in config.json
(swiglu_limit: 7.0, swiglu_limit: 10.0, activation_situ_beta: 4.0). The reports that give
a reason name loss spikes and overflow, and a bound that removes them changes the function and
the gradient wherever it bites.
Background: the feed-forward unit
Each Transformer layer has a token mixer (attention or one of its relatives), which moves information between positions, and a channel mixer, the feed-forward unit (FFN), which works on each token’s vector on its own. The 00 primer derives its cost: the FFN’s matrices hold most of a dense model’s parameters (64% in LLaMA-7B), and a matrix costs two floating-point operations (FLOPs) per weight per token in the forward pass. In a mixture-of-experts (MoE) model each expert is itself a small FFN; a router picks a few of them per token, and a shared expert, if there is one, runs for every token (C2). Everything here applies to one FFN, whether it is a dense layer, a routed expert or a shared expert.
The original FFN has two matrices and one activation function (ReLU, later GELU), applied entry by entry:
with model width and hidden width . We call the up projection and the down projection. With the unit has parameters and costs FLOPs per token. Its input is the output of an RMSNorm, so its size is set by the norm’s learned gains, not by the token.
Notation for this article: and are a hidden unit’s gate and up pre-activations (elsewhere in the series counts KV heads; no attention appears here), is the unit’s output before the down projection, and is a clamp limit.
Gated linear units
A gate made of the input
Dauphin et al. (2016) built a language model out of convolutions and borrowed the gate from the LSTM, which decides how much of each number to pass on by multiplying it with a sigmoid, a value between 0 and 1 computed from the input. Their gated linear unit (GLU) does this inside one layer: one projection of the input is the value, a second projection squashed by a sigmoid is the gate, and the output is their entrywise product ():
Their argument was about gradients. Write and . For the LSTM-style unit , which they call the gated tanh unit (GTU), the gradient is ; for the GLU it is . The GLU’s first path has no factor , which is near zero wherever tanh is flat, so gradients reach the value branch scaled only by the gate: “a multiplicative skip connection which helps gradients flow through the layers” (§3). At comparable parameter counts their GLU networks reached lower perplexity than GTU, tanh and ReLU networks.
Shazeer’s variants
Shazeer (2020) put GLUs into the FFN of a Transformer, with a third matrix and a choice of gate function :
The sigmoid gives GLU, the identity gives Bilinear, ReLU gives ReGLU, GELU gives GEGLU, and Swish, , with , , gives SwiGLU. The primer’s code writes the same unit with (gate), (up) and (down).
To compare at equal cost, Shazeer reduced “by a factor of 2/3”, from 3,072 to 2,048 in a T5-base model with . From a baseline that is , and : the same parameters and FLOPs as the two-matrix unit (computed; the primer shows LLaMA’s version of the rule).
two-matrix FFNdff = 4d = 16,384
gated FFNdff = 8/3·d ≈ 10,923
dashed: a third matrix without narrowing, three at 4d: 12d² = 201.3M, unmatched: 1.5×. Each matrix is drawn d = 4,096 tall and dff long on one scale (Wd as its transpose), so area is weights. φ is the activation; ⊙ multiplies the gate and up outputs entry by entry.
Table 1 of the paper reports held-out log-perplexity after 65,536 and 524,288 training steps, with all models matched for parameters and computation:
| FFN | 65,536 steps | 524,288 steps |
|---|---|---|
| ReLU (baseline) | 1.997 | 1.677 |
| GELU | 1.983 | 1.679 |
| Swish | 1.994 | 1.683 |
| GLU | 1.982 | 1.663 |
| Bilinear | 1.960 | 1.648 |
| ReGLU | 1.953 | 1.645 |
| GEGLU | 1.942 | 1.633 |
| SwiGLU | 1.944 | 1.636 |
The three ungated activations end within 0.006 of each other. Every gated variant beats all three, Bilinear included, which has no activation at all. (Dauphin’s convolutional model ranked Bilinear well below GLU; the two settings differ in architecture and in parameter matching, so neither result ranks the bilinear form in general.) GEGLU, not SwiGLU, has the lowest number at both step counts, by less than the spread Shazeer measured between repeated short runs of SwiGLU (0.010); his wording is that “the GEGLU and SwiGLU variants produce the best perplexities”. Fine-tuning results mostly favour the gated variants too, with the caveat “the results are noisy”. The paper ends: “We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.”
LLaMA used SwiGLU, and the map shows it stuck. Thirty of the 35 checkpoints pass this site’s
swiglu code check, which looks for act(gate(x)) * up(x) with a SiLU-type activation in the
modeling code. Gemma 4 uses GEGLU (hidden_activation: "gelu_pytorch_tanh"), as Gemma 1 and 2
said they did. Kimi K3 uses its own gate, SiTU-GLU, described below. The three NVIDIA Nemotron
checkpoints drop gating and use a two-matrix FFN with squared ReLU (mlp_hidden_act: "relu2").
What the product computes
Take one hidden unit, with and , and output . For large positive , and , a quadratic form in the input. For large negative the sigmoid closes the gate: SiLU never goes below (at , computed), so only the gate factor’s positive side is unbounded. The output still grows with on both sides.
An ungated unit is a ridge function: it changes only along the one direction . A gated unit depends on two directions and computes their product exactly; the FFN’s output is then a weighted sum of such products. Multiplicative units of this kind are old (the sigma-pi units of Rumelhart, Hinton and McClelland, 1986), and Jayakumar et al. (2020) show that multiplicative interactions enlarge the set of functions a layer of a given size can represent. Whether that is why gating wins in language models no source we read establishes; the strongest evidence in Shazeer’s table is the Bilinear row, whose only nonlinearity is the product.
Why the product makes outliers
Growth comes from the weights
The input to the FFN is normalized, so what grows during training is the weights. If one unit’s gate and up weights both grow by a factor , its output on a fixed input grows by where the gate is open, while a ReLU unit’s grows by . Along , SwiGLU gives 0.73 at , 15.7 at , 64.0 at and 1,024 at (computed); GELU gives about .
How large the product can get depends on how the two weight vectors point. For any vectors and , with , the symmetric part (the antisymmetric part contributes nothing to ). has rank two, with eigenvectors and eigenvalues . So over inputs of a fixed norm,
where is the angle between and , and the worst input points halfway between them. After an RMSNorm with gains , . Aligned weights (or anti-aligned ones, ) double the ceiling of orthogonal weights, and they make large gate values and large up values arrive on the same inputs.
Fishman et al. argue that training produces this alignment. Their Theorem 1 considers a stationary point of the loss with an penalty on the weights (a point where the gradient of loss plus penalty is zero) at which one unit’s gate sigmoid is saturated, flat at 1 on every training token. There, the gate and up weights satisfy and for one symmetric matrix , which they read as . (The equations force equal norms; exact alignment needs one more condition on .) The penalty picks the direction; weak regularization lets the norms grow, which they say strengthens the effect. In their Llama2 7B run, trained with AdamW, the correlation between the two weight vectors of an outlier channel “is initially low, but then it increases drastically between 125B and 210B tokens”, together with further norm growth, and the outliers appeared after about 200 billion tokens. They saw strongly negative correlations in other channels too.
|silu(g)·u|, relative to the largesttop 1%
value at s = 1 (shown once s > 1)
- GELU since s = 1
- ×1.0
- SwiGLU since s = 1
- ×1.0
- largest ÷ median |silu(g)·u|
- 71×
2,000 synthetic tokens: p, q standard normal (seeded), g = s·p, u = s·(ρp + √(1−ρ²)·q). GELU in its tanh form.
Where the large values land
The product is the input to the down projection, and that is where Fishman et al. found the trouble: “the SwiGLU activation function can lead to outliers in the input of the last linear layer of the MLP component.” Keeping only that tensor in higher precision made their FP8 run converge. Yang et al. (2024) found the same place from the inference side: quantizing LLaMA-2, LLaMA-3, Mistral, Mixtral and Gemma activations to 8-bit integers after training, they saw “activation spikes” at the input of the FFN’s down projection, in “the early and late layers”, and “dedicated to a couple of tokens”.
Yu et al. (2024) traced the residual stream’s “massive activations” (Sun et al., 2024: a few entries thousands of times larger than the rest) back to this product, amplified by a single large weight, the “super weight”, in an early layer’s down projection.
What a large value does to a low-precision number
FP8 E4M3 stores a number in eight bits: a sign, four exponent bits and three mantissa bits. Its largest finite value is 448. Its normal numbers run from to 448, a ratio of about 28,700, with eight steps between one power of two and the next, so rounding changes a value by at most 1/16 of itself anywhere in that range; below , precision thins out until values round to zero (computed from the bit layout). To cast a tensor to FP8, it is multiplied by a scale that puts its largest values near 448.
Fishman et al. used delayed scaling: one scale per tensor, taken from the largest values of earlier steps. Their failure was the delay. A value that has grown since the scale was set lands above 448 and is clipped, which they describe as a spike that “disrupts” the continuity delayed scaling assumes. That is the setting where one outlier does the most damage.
The labs in the map scale at a finer grain. DeepSeek-V4 trains with FP8 in blocks of 128×128 for weights and casts the SwiGLU output to FP8 right before the second expert matrix multiplication (its Fig. 5 shows “SwiGLU + FP8 Cast” between the two GEMMs); Kimi K3 stores most activations with block-wise FP8. With a scale per block of 32 to 128 values computed when the block is cast, an outlier affects only its own block, in one token’s row. Within that block, E4M3 still keeps every value larger than 1/28,700 of the outlier at full relative precision, so a block-scaled FP8 cast tolerates outliers well.
FP4 E2M1 has the nonzero magnitudes 0.5, 1, 1.5, 2, 3, 4 and 6, a ratio of 12. In a block scaled so its largest value lands at 6, every value less than 1/24 of that largest one rounds to zero (computed). The MX formats (MXFP4, MXFP8) share one power-of-two scale per block of 32 values. 8-bit integers, Yang et al.’s setting, have a uniform grid and lose small values the same way.
31 typical values, 0.05 to 1outlierto the value it rounds to
- scale
- ×224.0
- values clipped
- 1
- below E4M3's normal range
- 0
- rounded to zero
- 0
- worst relative error, typical values
- 5.9%
Delayed scale 448 / 2: set when the block's largest value was 2. Current scales 448 / outlier (FP8) and 6 / outlier (FP4), not rounded to powers of two. Round to nearest, ties to even.
Smooth-SwiGLU (Fishman et al.) is a response at this level that leaves the function alone. It multiplies channel of the up branch by a factor before the FP8 cast and folds into column of the down projection, so and the network computes the same thing at “zero additional cost at inference”. On Llama2 7B it reached results “on par” with BF16.
The bounds the models in the map ship change the function instead. The reports that give a reason name stability. DeepSeek-V4: “the occurrence of spikes is consistently tied to outliers in the MoE layers, and the routing mechanism itself appears to exacerbate the emergence of these outliers” (§4.2.3). Kimi K3: at extreme expert sparsity its routed path is “a chain of nearly four consecutive matrix multiplications” that “produces exploding internal activations”, and “both multiplicative factors in SwiGLU are unbounded, so coincident large coordinates can produce activation outliers and increase overflow risk in low-precision arithmetic” (§2.3). Neither ties its bound to a specific format.
Clamped SwiGLU
gpt-oss: clamp, sharpen, and add one
The gpt-oss model card says only this, in a footnote to §2.2: “Our SwiGLU implementation is
unconventional, including clamping and a residual connection.” The modeling code
(GptOssExperts._apply_gate) says the rest. Gate and up are stored interleaved in one matrix,
gate in the even columns and up in the odd ones:
gate, up = gate_up[..., ::2], gate_up[..., 1::2]
gate = gate.clamp(min=None, max=self.limit)
up = up.clamp(min=-self.limit, max=self.limit)
glu = gate * torch.sigmoid(gate * self.alpha)
gated_output = (up + 1) * glu
with swiglu_limit: 7.0 in the config and from the config class’s default. With
:
The gate is clamped from above only, since the sigmoid already bounds it below; the up branch is clamped on both sides, because it enters linearly. The gate function is the standard sigmoid approximation of GELU, so this gate is closer to GEGLU’s than to SwiGLU’s (our reading; the code doesn’t say). And the splits the output into : a gated unit plus an ungated GELU-like unit that shares the gate’s weights. That is presumably the footnote’s residual connection.
The output lies in (computed: ). The and the sharper gate also change the unit below the clamp: at it outputs 1.69 where SwiGLU outputs 0.73, 2.3 times as much (computed). The code check, which looks for a SiLU-type gate times an up projection, counts it as SwiGLU.
MiniMax-M3 and M3.1-Flash run the same formula, with a code comment that says so (“same as GPT
OSS, but the weights are not interleaved”) and both values in the config (swiglu_alpha: 1.702,
swiglu_limit: 7.0), and apply it to routed experts, the shared expert and the three dense
layers; their documents never mention it.
DeepSeek-V4 and the plain clamp
DeepSeek-V4’s report states its clamp’s limits and its reason: “we clamped the linear component of SwiGLU to the range of , while capping the upper bound of the gate component at 10”, which “effectively eliminates outliers and substantially aids in stabilizing the training process, without compromising performance” (§4.2.3). The DeepSeek-V4.1 report repeats “clamping … at a threshold of 10” without a reason. The report cites gpt-oss for the idea, but the code keeps SwiGLU itself, with no and no :
This is SwiGLU evaluated at the nearest point of the box , : unchanged inside it, and outside it the clipped coordinate stops mattering. The output lies within (computed). Hy4-preview’s and GLM-5.3-Flash’s modeling code implement the same clamped unit, each with the comment “Simple swiglu instead of alpha”.
The cost shows in the gradient. Where , the output no longer depends on , so no gradient reaches the gate weights through that unit for that token; where , the same holds for the up weights. When both are clipped, the unit sends nothing back to its input or to its gate and up weights, though its column of the down projection still trains. A straight-through estimator would pass the gradient as if the clamp were not there; these models don’t. (This is activation clipping, unrelated to gradient clipping, which rescales the whole gradient.) A unit clipped on many tokens stops learning from them, as a dead ReLU does.
Nothing trains the limit itself: is a constant in the config. Weight decay supplies a restoring force, though. A clipped token sends no gradient to the clipped weights, but decay keeps shrinking them, so a unit clipped on every token is pulled back towards the box, and the saturated stationary point of Fishman’s theorem is not available to it. That is our inference; whether it is why the clamp works, the reports don’t test. DeepSeek’s report lists the clamp among techniques whose “underlying principles remain insufficiently understood” (§6).
Step: the same clamp with smaller limits
StepFun’s checkpoints clamp after SiLU rather than before it,
. Since
, the two placements change the gate factor by at most
at , and the output by at most 0.045 (computed), so Step’s
unit is the plain clamp with a smaller limit. What differs is where it applies. In Step-3.7-Flash, swiglu_limits is 7 for the
routed experts of layers 43 and 44 (counting from 0, the last two of 45) and 0, meaning no clamp,
elsewhere; swiglu_limits_shared is 16 for the shared expert of the same two layers. The output
ceilings are and (computed). The documents don’t say why
those two layers were chosen; Yang et al.’s spikes concentrate in early and late layers, but
linking the two is our guess. In the next release, Step-5-preview, both lists are 7.0 for every
layer, so every FFN is clamped. (Its kernel is in a vLLM file, vLLM being a common serving
library, that the code check didn’t fetch; that it clamps after the activation like Step-3.7 is
our inference from its name, SwigluStepAndMul.)
- SwiGLU silu(g)·u
- plain clamp τ = 10: silu(min(g, τ))·clip(u, ±τ) (DeepSeek-V4, Hy4, GLM-5.3-Flash)
- plain clamp τ = 7 (Step's routed experts, within 0.045)
- gpt-oss τ = 7: (clip(u, ±7) + 1)·g′σ(1.702 g′), g′ = min(g, 7) (gpt-oss, MiniMax)
(g, u) plane, −20 to 20: plain τ = 10
a > 0a < 0clamps inactive
Plain clamp (DeepSeek-V4, Hy4, GLM-5.3-Flash), τ = 10: equals SwiGLU inside g ≤ 10, |u| ≤ 10. Output range −100.0 to 100.0; largest |a| = 99.995.
Who clamps what
A config field doesn’t say which FFNs it covers; that is visible only in the modeling code, and the labs differ:
| Checkpoints | Unit | Limit | FFNs clamped (from the code) |
|---|---|---|---|
| gpt-oss-120b | gpt-oss | 7 | all (it has only routed experts) |
| MiniMax-M3, M3.1-Flash | gpt-oss | 7 | routed, shared, dense |
| DeepSeek-V4-Flash, V4-Pro (5 checkpoints) | plain clamp | 10 | routed, shared (no dense layers) |
| DeepSeek-V4.1-Flash (2 checkpoints) | plain clamp | 10 | config and report only; not visible in the fetched code |
| Hy4-preview | plain clamp | 10 | routed only; shared expert and dense layer unclamped |
| GLM-5.3-Flash | plain clamp | 10 | routed, shared, dense, vision tower |
| Step-3.7-Flash | plain clamp, after SiLU | 7 / 16 | routed / shared, layers 43–44 only |
| Step-5-preview | plain clamp, after SiLU (inferred) | 7 | all layers |
The choice moves between releases of one lab: DeepSeek-V3.2 and Tencent’s Hy3 have no bound, DeepSeek-V4 and Hy4-preview do, and Z.ai bounds GLM-5.3-Flash but not GLM-5.3.
A clamp is part of the model
A checkpoint trained with a clamp has to be served with it. Where the clamp binds, removing it
feeds the down projection values it never saw in training, and the outputs change; how often it
binds, no report gives. The configs make it easy to get wrong. gpt-oss’s says
"hidden_act": "silu", which its expert code ignores, so a loader that trusts the field computes
plain SwiGLU and drops the clamp, the and the . MiniMax’s says "swigluoai", the
name of a serving kernel for the gpt-oss unit, which the config class overrides. DeepSeek-V4’s
code puts the clamp in _apply_gate “so the grouped_mm / batched_mm backends … apply the same
clamp + SiLU on top of their packed gate_up output instead of bypassing it”. Step-5’s serving code
refuses any other value (“Step5 fused MoE supports only swiglu_limit=7.0”). The layout is part of
it too: an interleaved gate and up (gpt-oss) loaded as two halves (MiniMax) pairs the wrong
columns.
SiTU-GLU: a smooth cap
Kimi K3 bounds the unit without a clamp. Its Sigmoid-Tanh Unit GLU (SiTU-GLU) applies a scaled tanh, , to the linear factor of the gate and to the up branch:
The config fields are activation_situ_beta: 4.0 for the gate and
activation_situ_linear_beta: 25.0 for the up branch, and the code computes the unit in float32.
K3 uses it in every FFN: the dense first layer, the two shared experts and the 896 routed experts.
The same tanh cap was used on attention logits before (Gemma 2 caps logits with
), and SiTU’s up branch is Dauphin’s GTU tanh with a temperature of 25:
Dauphin et al. removed that tanh to keep gradients flowing, and Kimi puts it back, scaled so it is
nearly linear where most values are and saturates only where the bound is meant to act.
Since and , every output satisfies (Appendix B). The cap bounds the gate factor’s positive side (its negative side stays closed by the sigmoid) and both sides of the up factor. The bound is approached slowly; at the output is 98.4 (computed).
Near zero, , so each factor is off by a term of third order, and the relative gap to SwiGLU is about . Along that gives the computed gaps: 0.5% at , 2.1% at , 7.8% at , then 24% at and 53% at , almost all from the gate’s ; the up factor with is 0.21% below linear at . The report’s claim that SiTU-GLU “matches SwiGLU to first order around the origin” is this expansion. As both go to infinity it becomes SwiGLU at every point, but for finite the two separate without limit.
The report’s case against hard clamping is the gradient: “Unlike hard clamping of gate pre-activations, the smooth cap preserves nonzero gradients away from saturation boundaries, which we find to give better training behavior” (Appendix B). The gradient is positive everywhere but small over much of the range. The derivative of is , which decays like . With the gate factor’s slope is 0.47 at , 0.07 at and 0.03 at (computed), where a clamp still passes SiLU’s full slope of about 1. The up factor’s slope is 0.86 at and 0.0013 at . A clamp passes the whole gradient up to its limit and none beyond; the smooth cap weakens it across the range and leaves a small tail, the old trade-off between hard-tanh and tanh. The report gives no ablation numbers for SiTU-GLU against SwiGLU or against a clamp; its scaling-efficiency gain over Kimi K2 is for all of K3’s changes together.
change β₁ and β₂
- SwiGLU
- SiTU-GLU β₁·tanh(g/β₁)·σ(g) · β₂·tanh(u/β₂)
- plain clamp τ = 10
- at s =
- 1.0
- SwiGLU
- 0.731
- SiTU-GLU
- 0.716
- clamp τ = 10
- 0.731
- SiTU vs SwiGLU
- −2.1%
- gate slope at g = s: SiLU
- 0.928
- clamp τ = 10
- 0.928
- SiTU gate
- 0.880
The cap’s other cost is arithmetic per element, which counts only where the matrix units are busy. SiLU needs an exponential and a division per hidden unit; SiTU-GLU adds two tanh evaluations. These run on the GPU’s special-function units, which have a small fraction of the throughput of the units that do the matrix multiplications. In decode the FFN is limited by reading weights, and the difference disappears. In training and prefill, where the matrix multiplications are compute-bound and run in FP8 or FP4, the activation is computed between the two matrix multiplications and can hold them up: DeepSeek-V4 proposes, as a suggestion to hardware designers, “replacing SwiGLU with a low-cost element-wise activation that involves no exponential or division operations” (§3.1). The clamps cost two comparisons.
Kimi K3 lists an RMSNorm among the same stabilizers. It sits outside the expert. K3’s experts run in a narrower latent space (3,584 instead of 7,168), and a shared projection maps their summed output back to full width; the RMSNorm normalizes that sum before the projection. C2 covers this latent MoE.
What a ceiling bounds
A bound holds for one hidden unit before the down projection. If every , the FFN’s
output satisfies : finite, but still free to
grow through the down projection’s weights, which is the route Yu et al.’s super weight takes. An
MoE layer adds its routing weights; DeepSeek-V4’s routed contributions sum with weight 1.5
(norm_topk_prob: true, routed_scaling_factor: 1.5). So a clamp removes two of the three
factors that can grow ( and ) and leaves the third.
What a known ceiling buys a quantizer is, in our reading, a scale that needs no history: with , a fixed FP8 scale of can never overflow and still represents values down to about 0.0035 at full precision (computed). Under fresh per-block scaling that matters less, which fits the reports citing stability rather than a format. A pre-activation clamp also bounds and themselves, which some training stacks keep in FP8 for recomputation; SiTU-GLU bounds only .
Comparison
| SwiGLU | gpt-oss unit | Plain clamp (DeepSeek, Hy4, GLM, Step) | SiTU-GLU | |
|---|---|---|---|---|
| Output | ||||
| Range of | unbounded | () | (), (), () | |
| Equal to SwiGLU | everywhere | only at | inside the box | to first order near 0 |
| Gate gradient | SiLU’s | full up to , zero beyond | full up to , zero beyond | reduced from (0.47 at 4, 0.03 at 10), never zero |
| Extra elementwise work | none | two clamps, a | two clamps | two tanh |
| Reason given in a report | quality (Shazeer) | none | MoE outliers and loss spikes (DeepSeek-V4) | activation explosion, overflow risk (K3) |
| Checkpoints in the map | 16 | 3 | 11 (2 config and report only, 1 inferred kernel) | 1 |
Two fixes leave the unit alone. Smooth-SwiGLU rescales around the quantizer, as above. And Qwen3.8-Flash-Next, a plain SwiGLU model, trained “without a single loss spike”, “without relying on explicit clipping methods such as qk-clip … or SwiGLU-clip”, crediting the sigmoid gates it puts on what attention, Gated DeltaNet and its Gated Residual write into the residual stream (T6, R2). The same gates let it store the residual state in FP8 at inference.
Where the models in the map sit
The 35 checkpoints split six ways on their FFN unit (from the modeling code, except DeepSeek-V4.1, whose two clamped checkpoints rest on config and report; counts are checkpoints, so DeepSeek’s five V4 checkpoints count five times):
| FFN unit | Checkpoints | Labs |
|---|---|---|
| SwiGLU, unbounded | 16 | Aleph Alpha, DeepSeek (V3.2), Liquid, Mistral, Poolside, Qwen (3), Tencent (Hy3), Thinking Machines, Xiaomi (4), Z.ai (GLM-5.2, 5.3) |
| SwiGLU, plain clamp | 11 | DeepSeek (V4 ×5, V4.1 ×2), Tencent (Hy4), Z.ai (GLM-5.3-Flash), StepFun (2) |
| gpt-oss unit, clamped | 3 | OpenAI, MiniMax (2) |
| SiTU-GLU | 1 | Moonshot (Kimi K3) |
| GEGLU, unbounded | 1 | Google (Gemma 4) |
| two-matrix ReLU², no gate | 3 | NVIDIA |
Seven of the 16 labs bound their unit in at least one checkpoint. The swiglu code check counts
30 present, because it reads the clamped units and the gpt-oss unit as SwiGLU and SiTU-GLU as
something else; the table splits that count by what the code computes.
The models’ FFN widths do not follow Shazeer’s ⅔ rule. Dense FFN layers (most MoE models keep one to three at the bottom) range from (GLM-5.2, GLM-5.3, MiniMax-M3 and M3.1-Flash) to (Kimi K3’s first layer, 33,792 at ). Two checkpoints land exactly on : MiMo-V2.5-Pro and V2.6-Pro, with 16,384 at . Six keep three matrices at the full , 50% more weights than the matched width: Gemma 4, Mistral Large 4, Laguna S 2.1, Inkling, MiMo-V2.5 and MiMo-V2.6-Flash. Experts are much narrower, from (Kolibri-1) to (gpt-oss-120b, Mistral Large 4).
Most bounded models ship low-precision weights (MXFP4 in gpt-oss, FP4 experts in DeepSeek-V4, FP8 in GLM-5.3-Flash and Step-5-preview), but that is storage, often added in post-training: DeepSeek-V4’s FP4 comes from quantization-aware training after pre-training, while its clamp ran “throughout the training”.
Easy to get wrong
- “SwiGLU won Shazeer’s comparison.” Gating won; GEGLU and SwiGLU tie within noise (Table 1).
- “A clamped SwiGLU is SwiGLU with a ceiling.” True of the plain clamp; the gpt-oss and MiniMax unit differs from SwiGLU everywhere (see “gpt-oss: clamp, sharpen, and add one”).
- “
swiglu_limitclamps the whole FFN.” Only what the code applies it to; see “Who clamps what”. - “
hidden_acttells you the activation.” For gpt-oss, MiniMax and Step it doesn’t; a loader that trusts it drops the clamp. - “Outliers come from large inputs.” The FFN’s input is normalized; the growth is in the weights and in how the gate and up weights align.
- “The bound protects FP8.” The reports name stability and overflow in general, and block-scaled FP8 tolerates single outliers well; four-bit formats and stale per-tensor scales are where one value does real harm.
- “A smooth cap passes more gradient than a clamp.” Only beyond the clamp’s limit; inside it the clamp passes SiLU’s full slope and the cap much less.
Check yourself
-
A model has . What hidden width gives a SwiGLU FFN the same parameter count as a two-matrix FFN of width , and how many parameters is that?
Answer
. Both have parameters, about 134 million. -
Compute the range of gpt-oss’s unit with , and say at which and its ends are reached.
Answer
The gate is capped at , giving . The up factor runs from to . So runs from about (for , ) to about 56 (for , ). -
In DeepSeek-V4’s unit, a token has and at one hidden unit. Which weights get a gradient through this unit for this token?
Answer
The gate is clipped (), so doesn’t depend on and the gate weights get nothing. is inside the box, so and the up weights get a gradient. The unit’s column of the down projection gets one too, since . -
Estimate SiTU-GLU’s relative gap to SwiGLU at without a calculator, and say which of the two caps causes most of it.
Answer
About . The gate’s cap, , gives ; the up branch’s only . -
A block of 32 values has one entry of 60 and the rest near 1. Scaled so that 60 lands at the format’s maximum, what happens to the 1s in FP8 E4M3 and in FP4 E2M1?
Answer
E4M3: the scale is , so the 1s become about 7.5, well inside the normal range, with at most 1/16 rounding error. E2M1: the scale is , so the 1s become 0.1, below half the smallest nonzero value (0.5), and round to zero.
In the map
Methods
- SwiGLU 2 sources
- SwiGLU clamping 2 sources
- SiTU-GLU 2 sources
- Sigmoid Tanh Unit (SiTU) 1 source
Taxonomy nodes
- dense feed-forward network 7 methods
Further reading
- Dauphin et al., Language Modeling with Gated Convolutional Networks (2016), arXiv:1612.08083. The GLU, its gradient argument against the GTU, and the learning curves in a convolutional language model.
- Shazeer, GLU Variants Improve Transformer (2020), arXiv:2002.05202. Five pages; the variants, the parameter matching, and the fine-tuning tables summarized here.
- Fishman et al., Scaling FP8 training to trillion-token LLMs (2024), arXiv:2409.12517. The alignment theorem and its assumptions, the late-training outliers, and Smooth-SwiGLU.
- Yang et al., Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs (2024), arXiv:2405.14428. Where spikes sit in trained models, and two inference-time fixes.
- Yu et al., The Super Weight in Large Language Models (2024), arXiv:2411.07191. How one weight in a down projection turns the GLU product into the residual stream’s largest activations.
- Jayakumar et al., Multiplicative Interactions and Where to Find Them (ICLR 2020). What multiplicative units add to a layer’s expressivity, with the classical lineage.
- OpenAI, gpt-oss-120b & gpt-oss-20b Model Card (2025), arXiv:2508.10925, §2.2. The one-line footnote; read it with the modeling code.
- DeepSeek-AI, DeepSeek-V4 technical report (2026), arXiv:2606.19348, §4.2.3, §3.1 and Fig. 5. The clamp, the loss-spike motivation, where the FP8 cast sits, and the case for cheaper activations. Use the PDF; the HTML loses the numbers.
- Kimi Team, Kimi K3: Open Frontier Intelligence (2026), §2.3 and Appendix B. SiTU-GLU’s definition, Figure 4, and the bound and first-order-match argument.
