Open Models Almanac

Residual & normalization · R2

Rethinking the residual stream

Wider residual streams and attention over depth lower the loss with little extra arithmetic in most designs; they cost activation bytes wherever many tokens run at once, and they differ in how they keep a learned weight on the identity path from compounding

Illustration of four narrow parallel water channels running down a series of shallow stone terraces. Small gates at each terrace let water cross between channels, and one wide funnel draws from all four.

A decoder layer of DeepSeek-V4-Flash reads its input from four vectors per token, where an ordinary transformer layer reads one. Its configuration says hc_mult: 4 and hidden_size: 4096, so between any two sublayers each token carries 16,384 numbers, 32,768 bytes in BF16 instead of 8,192. How the four vectors are combined is learned: before each sublayer one small matrix product computes 24 coefficients per token (4 weights for reading, 4 for writing and 16 for mixing the vectors among themselves). Those products hold 0.28% of the parameters a token uses in each layer (computed below), and DeepSeek reports that the arrangement costs 6.7% of training time by its measure.

The thing changed is the residual connection, xk+1=xk+fk(xk)x_{k+1} = x_k + f_k(x_k), which deep networks have used since ResNet. It makes each layer’s input the plain sum of the embedding and every earlier output, which the Kimi K3 report calls “a bottleneck reminiscent of RNNs over time”. In 2026 five labs in this site’s map shipped something else in at least one model: DeepSeek and Z.ai use manifold-constrained hyper-connections (mHC), Tencent a variant without the mixing, Qwen its Gated Residual, and Moonshot attention over depth. Of the 35 language models whose modeling code this site has read, 11 have one of these; the other 24 keep a single stream.

These designs lower the loss in their authors’ experiments, and most add under 1% to the arithmetic. They cost in two other places. The wider state must be read and written at every sublayer, which takes time when thousands of tokens are processed at once, as in training and in reading a long prompt. And a learned weight on the path that carries the signal through the depth can compound across layers.

What the residual stream is

The primer 00 has the decoder block in full. A block reads a vector xkx_k of width dd for each token, runs a sublayer fkf_k (attention or a feed-forward network) on a normalized copy, and adds the result back:

xk+1=xk+fk(RMSNorm(xk)).x_{k+1} = x_k + f_k(\mathrm{RMSNorm}(x_k)).

The vector xkx_k is the residual stream. R1 covers the norm; this article covers the addition. A model with LL decoder layers has K=2LK = 2L sublayers, one attention and one feed-forward per layer, and we index them by kk. We write dd for the width and nn for the number of parallel streams, and let fkf_k include its norm.

Unrolling from sublayer kk to any later sublayer k′k' gives (He et al. 2016)

xk′=xk+∑j=kk′−1fj,∂L∂xk=∂L∂xk′(I+∂∂xk∑j=kk′−1fj).x_{k'} = x_k + \sum_{j=k}^{k'-1} f_j, \qquad \frac{\partial \mathcal{L}}{\partial x_k} = \frac{\partial \mathcal{L}}{\partial x_{k'}}\Big(I + \frac{\partial}{\partial x_k}\sum_{j=k}^{k'-1} f_j\Big).

The signal from sublayer kk reaches k′k' with weight exactly one, the identity mapping, and the gradient at xkx_k contains the gradient at xk′x_{k'} unchanged, so it is “unlikely to be canceled out” however deep the stack.

He and colleagues argued that the weight should stay at one. With a scaled shortcut, xk+1=λxk+fkx_{k+1} = \lambda x_k + f_k, the unrolled sum begins with λk′−kxk\lambda^{k'-k} x_k: over the 86 sublayers of DeepSeek-V4-Flash, λ=1.01\lambda = 1.01 gives 2.35 and λ=0.99\lambda = 0.99 gives 0.42 (computed). On CIFAR-10, scaling the shortcut and branch of a 110-layer ResNet by 0.5 raised the error from 6.61% to 12.35%.

In a pre-norm transformer a constant scalar on the shortcut does less harm than this suggests, because every sublayer normalizes its input and RMSNorm ignores overall scale. Rescaling the stream by λ−k\lambda^{-k} shows the network computes the same function as a plain residual whose branch jj is weighted by λ−(j+1)\lambda^{-(j+1)}: at λ=1.01\lambda = 1.01, from 0.99 for the first sublayer to 0.42 for the last (our derivation, computed). A matrix on the shortcut can’t be absorbed this way: the norm removes only one common scale, and a product of matrices, even of one matrix repeated, also turns directions. (A fixed invertible mix can be moved into the read and write weights instead, which then carry the compounding.) That is the case the widened designs have to handle.

Qwen’s report gives the reason to change the residual at all, in the pre-norm network: “every block reads the same stream, so a feature written early must compete with everything written after it.” One way to see the options is to write the input of every sublayer as a weighted sum of earlier outputs,

inputk=∑i<kMi→k vi,\text{input}_k = \sum_{i<k} M_{i \to k}\, v_i,

with v0v_0 the embedding and viv_i the output of sublayer ii. For the plain residual every weight Mi→kM_{i \to k} is 1, a lower-triangular matrix of ones, whose blocks below the diagonal all have rank one. The Attention Residuals paper classifies the designs below by the structure they give MM (its §6.2), and we follow it.

Where the bytes go

A GPU computes in small, fast on-chip memory and stores tensors in large, slower main memory. A kernel is one program launched on the GPU; each kernel reads its inputs from main memory and writes its outputs back. Memory traffic is the bytes moved that way, and kernel fusion puts several operations into one kernel so the intermediate results stay on chip. A kernel that reads the whole residual state makes one pass over it.

Whether those bytes cost time depends on how many tokens run at once. Generating text (decode) produces one token per sequence per step; every step reads the weights the batch uses (in a mixture-of-experts model, the shared weights and the experts the tokens chose), and with few sequences that read takes most of the time. Reading a prompt (prefill) and training process thousands of tokens per step, so each weight byte is used thousands of times and the arithmetic dominates. Activation bytes behave the other way: they grow with the number of tokens. In DeepSeek-V4-Flash one layer reads about 209 MB of weights per decode step (experts at FP4, the rest at FP8), while its four-stream mHC layer, with the best fused kernel a V4 model can use (described below), adds 163,840 bytes of stream traffic per token over a plain residual. At one token that is 0.08% of the step’s bytes. The weight bytes stop growing once every expert is in use, which with 6 of 256 experts per token happens within a few hundred tokens, at about 3.35 GB per layer; the stream’s bytes keep growing with the tokens and reach that at about 20,000 tokens per step (computed, BF16 stream). So the stream’s bytes cost time in prefill, training and large-batch serving, and small-batch decode pays mostly in extra kernels run one after another, and in the residual mechanism’s own weights.

Pipeline parallelism splits the layers into consecutive groups (stages), one per group of GPUs, and sends each token’s residual state from one stage to the next; a wider stream sends more. Activation checkpointing, also called recomputation, keeps fewer intermediate results for the backward pass and recomputes the rest. Tensor parallelism splits each weight matrix across GPUs; the residual state is then usually copied on every GPU, and the sublayer’s output is summed across them (an all-reduce) before it is added back.

The wider state lives only while a token passes through the layers. The KV cache, the per-token memory attention keeps across steps, is unchanged by every design here, because attention and the experts still see one dd-wide input.

Widening with fixed weights

The simplest change keeps the layers and gives the stream more room. Replace the single vector by nn vectors per token, the streams or branches, start them as nn copies of the embedding, let each sublayer read a weighted sum of them, and write its output back. At the end the streams are combined into one vector for the output head.

Google’s AltUp (Baykal et al. 2023) did this for T5 models. Qwen tested how much width alone buys with a stripped-down version in a pre-norm decoder: each block holds nn learned scalars aia_i, reads ∑iaiRi\sum_i a_i R_i from the branches RiR_i, and adds its output to branch k mod nk \bmod n, round-robin by depth. That is nn extra parameters per block and no matrix product. On a 25B-parameter mixture-of-experts model with 3B active, trained on 400B tokens, it “lowers the training loss … by roughly 0.01” (Qwen3.8-Next report, §2.2).

In the language of MM, a sublayer’s weight on an earlier output now depends on which branch that output went to, so the weights repeat with period nn along the sources, and every block of MM below the diagonal has rank at most nn. Width is rank: nn streams keep up to nn independent combinations of the past instead of one sum.

  • fixed weight
  • learned, the same for every token
  • computed from the token
sublayer fattention or FFN1 stream↓ depthread: weight 1write: weight 14 streams↓ depthread: one learnedscalar per lane, summedℓ mod 4write: weight 1, intolane ℓ mod 4 only4 streams↓ depthread: static +tanh(token term)mix 4 × 4: static + tanhwrite: static +tanh(token term)4 streams↓ depthread: sigmoidmix 4 × 4: Sinkhorn-Knoppwrite: 2 × sigmoid4 streams↓ depthNNNNN: each lanenormalized. Read:sigmoid gate perchannel, rank-320bottleneck; meanwrite: 2 × sigmoid, onescalar per lane9 stored vectors at most (Kimi K3)embeddinglayers 0–11layers 12–23layers 24–35layers 36–47layers 48–59layers 60–71layers 72–83running sum, 84–92read: softmax oversourceswrite: weight 1, into therunning sum

Constraint: none: both weights are fixed at 1.

Figure 1.One sublayer boundary in six designs; the sections below explain each. A plain residual reads and writes with weight one. Widening adds lanes; hyper-connections learn the read, the write and a mix between lanes; mHC bounds them; Gated Residual drops the mix and gates the read per channel; Block AttnRes replaces the sum by attention over stored block summaries. Tencent's iHC is the mHC picture without the crossbar.

Width costs memory traffic, not arithmetic; in Qwen’s words, “the extra cost is the memory traffic of carrying nrn_r branches instead of one” (nrn_r is our nn). With a fixed read one pass suffices, reading the nn streams and the sublayer output and writing one stream and the next input: about 7d per sublayer at n=4n = 4 against 4d for a plain residual (computed, in the accounting of the traffic chart below). The form of the weights matters too: on a 1B-parameter OLMo model trained on 500B tokens, Zhu et al. found AltUp with two blocks ended slightly worse than the baseline (validation loss 2.827 against 2.811), while static hyper-connections, the next design with its coefficients held fixed, ended better (2.799 at two streams, 2.791 at four).

Hyper-connections: learning the read, the write and the mix

Hyper-connections (HC; Zhu et al. 2024, ByteDance) make all three operations learned and let them depend on the token. Stack the nn streams of one token as the rows of Xk∈Rn×dX_k \in \mathbb{R}^{n \times d}. A sublayer then computes

Xk+1=HkresXk+(Hkpost)⊤fk ⁣(HkpreXk),X_{k+1} = H^{\mathrm{res}}_k X_k + (H^{\mathrm{post}}_k)^\top f_k\!\left(H^{\mathrm{pre}}_k X_k\right),

where Hkpre∈R1×nH^{\mathrm{pre}}_k \in \mathbb{R}^{1 \times n} is the read (the sublayer’s dd-wide input as a weighted sum of streams), Hkpost∈R1×nH^{\mathrm{post}}_k \in \mathbb{R}^{1 \times n} the write (how much of the output each stream receives) and Hkres∈Rn×nH^{\mathrm{res}}_k \in \mathbb{R}^{n \times n} the mix. The sublayer still sees one vector of width dd, so attention and the experts are unchanged.

Each map is a learned static part plus a token-dependent part: a small linear map of the normalized streams, passed through tanh⁡\tanh and scaled by a learned factor that starts at 0.01. At initialization the streams are copies of the embedding, the read takes one stream, the write adds the output to all of them and the mix is the identity. Every stream then receives the same updates, so each one equals the pre-norm transformer’s residual until training moves the coefficients.

On OLMoE-1B-7B (1.3B active of 7B, 500B tokens) four streams lowered the training loss by about 0.027 and the C4 validation loss by 0.028, and the authors report reaching the baseline’s loss 1.8 times sooner. On a 7B dense model the validation loss fell from 2.322 to 2.304, and the baseline showed “frequent spikes” while the HC model showed none. Arithmetic rose by 0.15 to 0.21%.

Activation memory in training rose more: by 26% for a 1B model and 28% for the 7B model at four streams, measured on eight GPUs, and by 9.7% for the mixture-of-experts OLMoE. Static HC cost the same 26% as the token-dependent version, so the cause is the four-times-wider state each sublayer keeps for the backward pass, not the coefficients. HC reports no wall-clock time and no memory-traffic or pipeline numbers.

mHC: bounding the product

Unroll HC’s update and the term that carries XkX_k forward to sublayer k′k' is an ordered product of mixes, Hk′−1res⋯HkresH^{\mathrm{res}}_{k'-1} \cdots H^{\mathrm{res}}_{k}. If the factors drift from the identity, the product’s gain along some direction can grow or shrink geometrically in the number of factors; that is what compounding means here. Xie et al. (DeepSeek) measured it on a 27B model, with the largest absolute row sum of the product as the forward gain and the largest column sum as the backward gain, and found “peaks of 3000” for HC, with a loss surge around step 12,000 that tracked the gradient norm.

mHC constrains the mix to doubly stochastic matrices: nonnegative entries, every row and every column summing to one. Each such matrix is a weighted average of permutation matrices (the Birkhoff–von Neumann theorem), so the mix can only shuffle and average the streams. A product of two such matrices is again one, since AB1=A1=1AB\mathbf{1} = A\mathbf{1} = \mathbf{1} and likewise for columns. Each output stream is a weighted average of the inputs, so the forward signal can’t grow; the backward pass multiplies by the transpose, whose rows are the columns, so neither can the gradient. Together these make the spectral norm exactly 1 (at most 1 from the two sums, and the all-ones vector attains it). With n=1n = 1 the only such matrix is the number 1, the plain residual.

What this does to the streams can be said exactly (our derivation). Because the columns sum to one, the mean of the streams, mk=1n1⊤Xkm_k = \tfrac{1}{n}\mathbf{1}^\top X_k, follows

mk+1=mk+hˉk fk,hˉk=1n∑iHk,ipost,m_{k+1} = m_k + \bar h_k\, f_k, \qquad \bar h_k = \tfrac{1}{n}\textstyle\sum_i H^{\mathrm{post}}_{k,i},

a plain residual stream with a bounded write weight. The differences between the streams are multiplied by the mixes, which can only shrink them, and refilled by unequal writes: they act as n−1n - 1 fading memories of recent outputs. How fast they fade depends on how far the mixes are from permutations. A mix whose entries are all at least ε\varepsilon shrinks the spread between the streams by a factor of at least 1−nε1 - n\varepsilon per sublayer (our derivation), so with every entry at 0.05 or more the carried part is within 5% of copies after about 14 sublayers; mixes close to permutations keep the streams apart.

draw 1 of 20
  • unconstrained I + εG (band: 20 draws)
  • softmax only, no Sinkhorn
  • Sinkhorn, as in DeepSeek-V4's code (mHC)
largest gain of the product (log scale)10−1110102103120406080100120depth: number of multiplied 4 × 4 matricesgain 1depth 86TOY: random 4 × 4 matriceslargest gain of the product (log scale)10−1110102103120406080100120depth: number of multiplied 4 × 4 matricesgain 1depth 86TOY: random 4 × 4 matricesunconstrainedsoftmax onlySinkhorn (mHC)

Toy, at depth 86: unconstrained I + 0.10·G 5.26 (20 draws: 3.40 to 21.3), softmax only, no Sinkhorn 1.64, Sinkhorn with 20 iterations 1.02. Gain = the larger of the largest absolute row sum and column sum of the product.

How different the 4 streams can still be: second singular value of the Sinkhorn product σ2 = 9.0 × 10−12 at depth 86 (the largest is 1; σ2 near 0 means the product has averaged the four streams into one).

Figure 2.A toy: products of random 4x4 mixing matrices. Unconstrained matrices that start near the identity drift away from gain 1 as they compound, more with depth and with their distance from the identity. Sinkhorn-rescaled matrices stay at gain 1 forward by construction, and backward up to the truncation error, which shows only for sharp mixes and few iterations. The readout shows the other side of the bound: the streams are pulled towards their mean.

To land in the set, mHC exponentiates the raw coefficients and alternately rescales rows and columns to sum to one, the Sinkhorn–Knopp iteration (1967). For a strictly positive matrix this converges to the unique doubly stochastic matrix of the form D1eZD2D_1 e^{Z} D_2, with D1,D2D_1, D_2 diagonal (an entropic projection), slowest for mixes close to a permutation. DeepSeek stops after 20 (hc_sinkhorn_iters: 20), and the gradient flows through all 20 steps, so the model trains with the same truncated matrix it runs with.

Which sums come out exact depends on the last step. The Transformers port of DeepSeek-V4 (DeepseekV4HyperConnection) starts from a row softmax, ends on a column normalization and applies the transpose, so in the convention of the equation above every row of the mix sums to one, exactly in FP32, and the forward gain of any product is 1; the port casts the mix to BF16 before applying it, which moves a row sum by up to about 0.3% (computed). The columns sum to one only up to the truncation. That error is the backward gain the mHC paper measures at “approximately 1.6” at most, against “nearly 3000” for HC, and it is also the error in the conservation of the stream mean. These are token averages; collected token by token in their own re-implementation on models up to about 0.36B parameters, Yang and Gao (mHC-lite, 2026) found many column sums far from 1.

The read and the write get a lighter bound: a sigmoid for the read, which keeps its weights positive (the sublayer’s norm removes their overall scale), and twice a sigmoid for the write, between 0 and 2. Both prevent “signal cancellation” from mixed signs.

On the 27B model mHC ended with a loss 0.021 below the baseline’s and scored above HC on seven of eight benchmarks: BBH rose from 43.8 (baseline) and 48.9 (HC) to 51.0, and MATH was the exception at 26.4 for HC and 26.0 for mHC. HC still gained on every benchmark despite the spike; the paper’s case is that mHC keeps the gain without an instability that grows with scale. A forward gain that scales every stream alike is invisible to the normalized sublayers; one that grows along some directions changes what they read, and it shows up as early writes that drown later ones and as backward gain.

What mHC costs

In DeepSeek-V4-Flash one mHC module has a projection of (n2+2n)×nd=24×16,384(n^2 + 2n) \times nd = 24 \times 16{,}384 weights, plus 27 biases and gains: 393,243 parameters. There are two per layer, so 33.9M for the model including the output collapse (computed from the code shapes). That is 0.28% of the 284M parameters a token uses per layer, and the module’s arithmetic is 0.36% of the layer’s. The code keeps these weights in FP32 while the experts are FP4 and the rest FP8, so in decode they are 135.5 MB per step, 1.5% of the weight bytes at batch 1 (computed). In prefill a product with only 24 outputs uses the GPU’s arithmetic poorly, so its share of time exceeds its share of arithmetic.

The mHC paper counts what one layer of HC reads and writes per token outside the sublayer, unfused: (5n+1)d(5n+1)d reads and (3n+1)d(3n+1)d writes plus a few coefficients, against 2d2d and dd for a plain residual, 34d against 3d at n=4n = 4, about 11 times as much. Pipeline parallelism sends nn times the data between stages. DeepSeek’s V4 report lists three counter-measures: fused kernels for training and inference; recomputing most mHC intermediates in the backward pass instead of storing them; and an adjusted DualPipe, DeepSeek’s pipeline schedule that runs the transfers between stages while the GPUs compute. With all three, mHC takes “6.7% of the overlapped 1F1B pipeline stage” in wall time during training (1F1B is the schedule in which each stage alternates one forward and one backward micro-batch).

Under tensor parallelism the four streams are copied on every GPU of the group, each GPU repeats the coefficient product, Sinkhorn and the update, and the all-reduce of the sublayer output must finish before the write. The vLLM code for V4.1 fuses that all-reduce into the mHC kernel for one configuration (four GPUs, up to 16 tokens).

Single-pass mHC: one traversal per boundary

DeepSeek changed the model in V4.1-Flash to save one more pass. The V4.1 report counts the traffic per sublayer boundary in elements of width dd. The ideal reads the nn streams and the sublayer output once and writes the new streams and the next sublayer’s input once: (2n+2)d(2n+2)d, 10d at n=4n = 4, the same count that gives 4d for a plain residual at n=1n = 1. V4’s training implementation ran three mHC kernels plus the pre-norm and moved (4n+4)d(4n+4)d, 20d.

The obstacle is a dependency. The next sublayer’s read coefficients are a function of all ndnd numbers of the updated streams, through a norm and the projection. A kernel works through the streams one tile at a time, so it can’t apply the coefficients to the first tile before it has seen the last. One pass computes the coefficients and a second reads the ndnd stream values again to apply them: (3n+2)d(3n+2)d, 14d. Single-pass mHC removes the dependency by shifting the read by one step: each sublayer reads with coefficients computed by the previous one, which are known before the pass starts. V4.1 also drops V4’s learned output collapse in favour of the last read. With the shift, the serving kernel Mega-mHC reaches the 10d bound. DeepSeek keeps the older multi-kernel code for pre-training, “since the shift only changes which mixing coefficients each block applies”, so the model is pre-trained with the shift and served with the fused kernel; the report says the shift “incurs negligible performance degradation” without a number. Once fused, the mix itself costs no extra pass; the dependency does.

Other parts of V4.1 work on four streams too: its three draft layers for speculative decoding are full decoder layers with their own mHC, fed the main model’s stream mean, and its Engram module, a hashed n-gram memory, writes into each stream separately.

Tencent’s variant: no mix at all

Hy4-preview keeps four streams and DeepSeek’s form of the read and write, token-dependent, but has no mixing matrix and no Sinkhorn. Its documents call it iHC, “identity Hyper-Connections”; the code docstring says “Independent Hyper-Connection”. Each stream is updated as xk+1,i=xk,i+Hk,ipostfkx_{k+1,i} = x_{k,i} + H^{\mathrm{post}}_{k,i} f_k: the identity path of every stream is the identity, so there is no product to bound. hc_magnitude: 2.0 is the upper bound of the write gate, DeepSeek’s 2 made configurable. Without the n×nn \times n part a module has 196,618 parameters instead of the 589,851 an mHC module of the same width would have (computed).

Gated Residual: put the capacity in the read

Qwen ran an ablation in its own framework, on a 25B-A3B model trained on 560B tokens, to find where HC’s gain comes from. All variants used four branches.

Residual designLossBenchmark average
Pre-norm, one stream1.61750.91
mHC, static coefficients1.59652.49
mHC, token-dependent coefficients1.59454.47
Gated Residual1.59054.66

The static widened stream takes 0.021 off the loss; making the coefficients depend on the token takes 0.002 more but adds 1.98 benchmark points. A sigmoid beat tanh⁡\tanh. Refining the read from one scalar per branch to one weight per branch and channel helped, while the same refinement of the write “gives almost nothing”. And once the read and write were expressive, “adding the nr×nrn_r \times n_r mixing operator brings no significant improvement.”

Gated Residual (GR), in Qwen3.8-Flash-Next (6B active parameters), is built from those findings. With R^i\hat R_i the branch RiR_i after its own RMSNorm and R^\hat R all four stacked,

u=1n∑i=1nσ(Gi)⊙R^i,G=W↑ SiLU ⁣(1nW↓R^),Ri′=Ri+2σ(si) f(u),u = \frac{1}{n}\sum_{i=1}^{n} \sigma(G_i) \odot \hat R_i, \quad G = W_{\uparrow}\,\mathrm{SiLU}\!\big(\tfrac{1}{n} W_{\downarrow} \hat R\big), \quad R_i' = R_i + 2\sigma(s_i)\, f(u),

where the gate GG has one entry per branch and channel, W↓W_{\downarrow} and W↑W_{\uparrow} form a bottleneck of rank 320 over the 4×2,560=10,2404 \times 2{,}560 = 10{,}240 stacked channels (hc_count: 4, hc_lowrank: 320), and ss is one write scalar per branch, also predicted from R^\hat R. There is no mix, so every branch keeps an identity path and nothing compounds. The read is the GatedNorm of R1 applied to the widened stream, and a squeeze-and-excitation gate (Hu et al. 2017) whose squeeze is over the token’s own branches instead of over image positions, with a reduction ratio of 32. Because it already normalizes, GR replaces the block’s pre-norm.

GR is the one design here with a visible arithmetic cost. A module has 6.6M parameters, almost all of them in the 2×320×10,2402 \times 320 \times 10{,}240 bottleneck, 27 times an mHC module at the same width. With two per layer and a read-only one before the output head, 640.6M parameters run for every token, about 11% of the 6 billion active parameters Qwen reports, and in BF16 they are 1.28 GB read per decode step (computed). In return GR matched Qwen’s own mHC in the ablation above. Qwen’s stress test on a 28-layer 25B-A3B model, with Muon at four times the optimal learning rate, found that “the configuration with the gated residual records zero loss spikes”, and adding GR to the Qwen3.5 structure lowered the loss at 276B tokens of the production configuration by 0.026.

By Qwen’s account, dropping the mix also removes “a full read of the residual state per block”. That holds against an unfused mHC. With a fused write, GR moves the same 14d as two-pass mHC (computed): its gate, like mHC’s read, needs a reduction over all branches before the read can start, the dependency DeepSeek removed with its shift.

Qwen stores the branches in FP8, because bounded gates in GR, in gated attention and in Gated DeltaNet “all bound the magnitude of what is written into the stream”; FP8 “halves the bytes moved for the residual state relative to BF16, with almost no loss in quality.” The report gives no number for the loss and no format or scaling scheme, and the pinned Transformers code keeps the stream in BF16. Qwen also trained a variant, from scratch and from mid-training, in which each block reads only the two branches with the highest gates. Pre-training loss and benchmarks were “almost unaffected”, but “the quality degraded clearly after post-training, so we did not adopt it”: “a case where pre-training metrics alone would have led to the wrong decision”.

Figure 3.Activation elements read and written per token at one sublayer boundary, at four streams, in one accounting (DeepSeek's three mHC counts from its V4.1 report, the rest computed). A plain residual moves 4d. Widening by four costs 10d at best, 14d with one extra pass, and 20d in DeepSeek's original training kernels; Block AttnRes in Kimi K3 moves about 6d. This is traffic per boundary, not the state held per token or the weights read per step.

Attention residuals: attention over depth

Moonshot took the other route. Instead of more width, attention residuals (AttnRes; Kimi Team 2026) give each sublayer a choice of what to read. In the plain residual sublayer kk reads the sum of everything before it; in AttnRes it reads a softmax-weighted average of the embedding and each earlier sublayer’s output:

uk=∑i=0k−1αi→k vi,αi→k=exp⁡ ⁣(wk⊤RMSNorm(vi))∑j=0k−1exp⁡ ⁣(wk⊤RMSNorm(vj)).u_k = \sum_{i=0}^{k-1} \alpha_{i \to k}\, v_i, \qquad \alpha_{i \to k} = \frac{\exp\!\big(w_k^\top \mathrm{RMSNorm}(v_i)\big)}{\sum_{j=0}^{k-1} \exp\!\big(w_k^\top \mathrm{RMSNorm}(v_j)\big)}.

The query wk∈Rdw_k \in \mathbb{R}^d is a learned vector, a “pseudo-query”, the same for every token; the weights still depend on the token through the keys. The keys are RMS-normalized so that a sublayer with large outputs can’t take all the weight; the values are not. In terms of MM, full AttnRes makes the weights dense and token-dependent.

Weighted sums over layers are old: ELMo mixed a network’s layers with learned softmax weights, DenseNet feeds every earlier output to each layer, and DenseFormer (Pagliardini et al. 2024) gives each block static learned weights over all earlier ones. AttnRes adds weights that depend on the token and a query per sublayer. In the paper’s ablation on a 16-layer model with 436M active parameters, DenseFormer’s static weights gained nothing (loss 1.767 against 1.766 for the plain residual), input-independent mixing reached 1.749 and AttnRes 1.737.

Every read is a convex combination, so its size is at most that of its largest source, where the plain sum grows with depth, and an earlier output reaches a later sublayer through a single weight, not a product of weights, so nothing compounds. The price is that no path keeps a weight fixed at one: an early output reaches a later sublayer only as strongly as that sublayer’s softmax weights it, and a source whose weights fall towards zero gets almost no gradient through those reads. The factor α\alpha is not itself a loss against pre-norm, because the norm after each read rescales the convex average (our derivation). The paper’s measured gradients are more uniform across depth than the baseline’s. With the pseudo-queries initialized to zero the weights are uniform, and since each sublayer normalizes its input, a uniform average gives exactly the pre-norm transformer’s inputs: training starts from the ordinary network (our derivation).

Full AttnRes costs O(K2d)O(K^2 d) arithmetic per token and O(Kd)O(Kd) memory to keep every output. The paper calls the arithmetic feasible for L<1000L < 1000 layers and the K3 report “affordable” at L<100L < 100. Memory limits the full form: ordinary training keeps those outputs for the backward pass anyway, but with recomputation and pipeline parallelism every output must stay alive and cross every later stage.

Block attention residuals cut that to the number of blocks. The sublayers are split into NN blocks; within a block the outputs are summed as in a plain residual, and each sublayer attends over the completed block sums, the embedding, and the running sum of its own block. Memory falls from O(Kd)O(Kd) to O(Nd)O(Nd), and the paper finds that “N≈8 recovers most of the benefit across model scales”. The block size is a fixed choice; the loss trains only the weights over the sources.

Sublayer:Variant:
  • source held for the read (embedding, a completed block's sum; in full, every earlier output)
  • running sum of the current block
  • the sublayer reading
embedding (its own source)0–111 sum12–231 sum24–351 sum36–471 sum48–5960–7172–8384–929Block attention residuals: 6 sources for this readEach cell is one layer: attention on top,feed-forward below. The last group holds 9layers and never closes.emb.1 sum1 sum1 sum1 sum0–1112–2324–3536–4748–5960–7172–8384–92Block attention residuals: 6 sources for this readEach cell is one layer: attention on top, feed-forward below. The last group holds 9 layersand never closes.

Layer 50, attention. Block: sources read: 6 (of 9 at most): the embedding, the sums of layers 0–11, 12–23, 24–35, 36–47, and the running sum of layers 48–49. Held during the forward pass, per token in flight: 6 vectors = 86,016 bytes at BF16. Full: 101 sources (up to 187 at the output), held per token in flight: 101 vectors = 1,447,936 bytes at BF16.

Figure 4.Kimi K3's 93 layers in blocks of 12, the last holding 9. A sublayer reads the embedding, every completed block's sum and the running sum of its own block: at most 9 vectors per token, held only while the token passes through. Full attention residuals would read every earlier sublayer's output, up to 187.

Kimi K3 has 93 layers and attn_res_block_size: 12. The report describes “8 blocks with 12-layer size, giving a partial final block and 9 total blocks when counting the embedding layer”: seven blocks of 12 and a last one of 9 (computed from the config). The code has a pseudo-query per sublayer, so every sublayer after the first and the output head read their own weighted average, 186 reads of 2 to 9 sources each. A token in flight holds at most 9 vectors of 7,168 values, 129,024 bytes at BF16, against 14,336 for a single stream; full AttnRes would hold 187, 20.8 times as many. Over a 128K-token prompt processed in one piece the 9 vectors take 15.75 GiB (computed); the Attention Residuals paper splits them across GPUs and processes the prompt in chunks. AttnRes adds 2.7M parameters and 0.02% of a layer’s arithmetic.

K3 handles these costs in its systems. It wraps the AttnRes computation in activation checkpointing, so the activations saved per layer are “identical to that of the standard residual architecture”, and under pipeline parallelism sends only newly completed blocks between stages (“cache-based pipeline communication”). At inference a two-phase schedule reads the stored block sums once per block in one batched pass, then folds in the running sum sublayer by sublayer with an online-softmax merge: two softmax averages over disjoint sources combine exactly from their outputs and their running maximum and sum of exponentials. In prefill K3 splits the tokens across the tensor-parallel GPUs so each token’s block sums live on one GPU; in decode it fuses the merge into the all-reduce. Its EAGLE-3 draft model reads the outputs of the 1st, 4th and final AttnRes blocks. The Attention Residuals paper reports under 4% training overhead with pipeline parallelism and under 2% inference latency “on typical inference workloads”; the K3 report gives no overhead number.

In the paper’s scaling runs (194M to 528M active parameters), Block AttnRes reached the baseline’s loss with what the authors call a 1.25 times compute advantage, and its mHC baseline, mHC-lite, stayed within 0.006 of it at every size: lower at the three smallest (194M to 296M) and 0.001 higher at the two largest. On the paper’s 48B Kimi Linear model Block AttnRes matched or beat the baseline on every benchmark listed, GPQA-Diamond from 36.9 to 44.4. In Qwen’s runs on a 28-layer model, full AttnRes reached 1.762 without GatedNorm, level with GR’s 1.762 (GR’s read already carries the gate), and 1.758 with it, and Block AttnRes with four sublayers per block reached 1.768 with GatedNorm; K3 sums 24 sublayers per block. Moonshot’s “approximately 2.5×” scaling efficiency of K3 over K2 is credited to four changes together, AttnRes among them.

The paper’s 5.5d per layer for Block AttnRes against 34d for mHC compares with the unfused mHC count above and counts 3d for a plain residual; in the chart’s accounting K3’s Block AttnRes moves about 6.3d against 4d.

Side by side

DesignIdentity pathDepth weights MMConstraintExtra parametersElements per boundary, n=4n=4Reported gain (scope)Shipped in
Plain residualweight 1all ones, rank 1none needed04dbaseline24 of 35
Width only (simplified AltUp)identity per streamperiodic, rank ≤ nnonenn per block7d (computed)loss −0.01 (Qwen, 25B-A3B, 400B tokens)none
HClearned n×nn \times n productrank ≤ ntanh⁡\tanh on the token part only0.03% (OLMo-1B)34d unfusedloss −0.027 (OLMoE-1B-7B)none
mHCdoubly stochastic productrank ≤ nsigmoid read, 2σ write, Sinkhorn mix33.9M, FP32 (V4-Flash)20d → 14d → 10dloss −0.021, BBH +7.2 (27B)DeepSeek V4, V4.1; GLM-5.3-Flash
iHCidentity per streamrank ≤ nsigmoid read, gated write30.8M (Hy4-preview)not publishednot publishedHy4-preview
Gated Residualidentity per streamrank ≤ nsigmoid gates, per-branch norm640.6M (Qwen3.8-Flash-Next)14d (computed)loss −0.027, average +3.75 (Qwen, 25B-A3B)Qwen3.8-Flash-Next
Block AttnResone convex weight per sourcebetween N and N + Ssoftmax, normalized keys2.7M (Kimi K3)about 6.3d (computed)1.25× compute (194M–528M active)Kimi K3

Where the models in the map sit

From the configs and modeling code in this site’s run, 39 models have a usable code check; 35 are language models, and the four robot models all keep one stream. No code check covers the residual stream yet, so this table is read from the code directly.

ModelDesignConfig fieldsLayers × widthFrom the code
DeepSeek-V4-Flash, -0731, -Vision-ExpmHChc_mult: 4, hc_sinkhorn_iters: 2043 × 4096two modules per layer; learned weighted collapse at the end
DeepSeek-V4-Pro, -0813mHCsame61 × 7168same code as V4-Flash
DeepSeek-V4.1-Flash, -1017single-pass mHCsame40 × 5120read shifted by one sublayer; no learned collapse; draft layers carry four streams
GLM-5.3-FlashmHCsame, plus "mhc": true45 × 4096DeepSeek’s class line for line; plain mean at the end
Hy4-previewiHChc_mult: 4, hc_magnitude: 2.0, enable_ihc: true78 × 6144no mix, no Sinkhorn
Qwen3.8-Flash-NextGated Residualhc_count: 4, hc_lowrank: 32048 × 2560replaces the pre-norms; a read-only GR before the output head
Kimi K3Block AttnResattn_res_block_size: 1293 × 71688 stored entries plus a 9-layer partial block

All 11 come from five labs, and the same labs’ earlier models in the map (DeepSeek-V3.2, GLM-5.2 and GLM-5.3, Hy3, Qwen3.5 and Qwen3.6) use a single stream. No lab that keeps a single stream says it tried a widened or attention-based residual and rejected it. Nobody has published a head-to-head of mHC, Gated Residual and AttnRes at production scale. The closest are Qwen’s runs, which compare its own versions of mHC and AttnRes with GR on 28- and 48-layer models and a 25B-A3B MoE.

Easy to get wrong

  • mHC does not keep the signal norm at one. The exact constraint keeps the stream mean a plain residual and shrinks the differences between streams; with DeepSeek’s 20 Sinkhorn steps the forward gain is 1 (exactly in FP32, up to BF16 rounding as applied) and the backward gain is off by the truncation, at most 1.6 in the mHC paper’s token averages, with larger per-token outliers in mHC-lite’s measurements of small models.
  • The wider stream does not cost decode time in proportion to its bytes. At small batch the step’s bytes are weights: the stream adds 0.08% in DeepSeek-V4-Flash, while mHC’s FP32 weights add 1.5%; in Qwen3.8-Flash-Next, GR’s weights are about 11% of the active parameters. The stream’s traffic costs time in prefill, training and large batches.
  • Single-pass mHC is a model change, not only a faster kernel: each sublayer reads with the previous sublayer’s coefficients. DeepSeek pre-trains with the shift and calls its quality cost “negligible” without a number.
  • Kimi K3 does not have 8 blocks of 12 layers. It has 93 layers: seven blocks of 12, a last block of 9, and the embedding as its own source.
  • Pre-training loss alone can miss what a residual change does after post-training: Qwen’s top-two-branch read was “almost free in pre-training loss” and was dropped for what it did later.

Check yourself

DeepSeek-V4.1-Flash has hidden_size: 5120 and four streams. How many bytes per token does its residual state take between two sublayers in BF16, and how many activation elements does one boundary move with single-pass mHC, against a plain residual?

The state is nd=4×5120=20,480n d = 4 \times 5120 = 20{,}480 values, 40,960 bytes in BF16, against 10,240 for one stream. Single-pass mHC moves (2n+2)d=10×5120=51,200(2n+2)d = 10 \times 5120 = 51{,}200 elements per boundary, 102,400 bytes; a plain residual moves 4d=20,4804d = 20{,}480. The model is the one in the map that was built for the shift; a V4 model would need the two-pass (3n+2)d(3n+2)d.

The identity path of DeepSeek-V4-Pro (61 layers) passes through how many mixing matrices, and why can’t their product amplify the forward signal?

Two mixes per layer, so 122. Each mix, as applied, has rows that sum to one (exactly, before the cast to BF16) with nonnegative entries, so each output stream is a weighted average of the inputs, and a product of such matrices keeps that property. The columns sum to one only up to Sinkhorn’s truncation, so the backward gain can drift slightly. HC’s free mixes have no such property; their product can grow or shrink geometrically, like He et al.’s ∏λ\prod \lambda.

Why does the fused Mega-mHC kernel need the one-step shift to reach 10d?

Without it, the next sublayer’s read coefficients depend on all ndnd values of the updated streams, and the kernel can’t apply them until it has read every tile, so it reads the streams a second time: (3n+2)d=14d(3n+2)d = 14d. With the shift the coefficients come from the previous sublayer and are known before the pass, so one read and one write suffice: (2n+2)d=10d(2n+2)d = 10d.

Kimi K3’s output head reads how many sources, and why is that fewer than the number of layers?

Nine: the 8 stored entries (the embedding and the seven completed sums of layers 0 to 83) and the running sum of layers 84 to 92. Within a block the outputs are summed as in a plain residual, so the model holds one vector per block instead of one per sublayer.

In the map

Methods

Taxonomy nodes

Further reading

  • He et al., Identity Mappings in Deep Residual Networks (2016), arXiv:1603.05027: the unrolled residual and the scaled-shortcut argument.
  • Zhu et al., Hyper-Connections (2024), arXiv:2409.19606: the original design, its results and its memory cost.
  • Baykal et al., Alternating Updates for Efficient Transformers (2023), arXiv:2301.13310: AltUp’s predict-and-correct widening on T5.
  • Xie et al., mHC: Manifold-Constrained Hyper-Connections (2025), arXiv:2512.24880: the gain measurements, the constraint, and the kernel and pipeline design.
  • DeepSeek-V4 report (arXiv:2606.19348), §2.2 and §3.4.2, for mHC as shipped and its 6.7% cost; DeepSeek-V4.1 report §2.4.1 for single-pass mHC and the traffic accounting the chart uses.
  • Qwen, On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability (arXiv:2608.30320), §2.2 and §3.3: the ablation, Gated Residual and the stability tests.
  • Kimi Team, Attention Residuals (2026), arXiv:2603.15031: full and block AttnRes, the depth-weight view (§6.2) and the systems design; the Kimi K3 report §2.2 for the shipped configuration.
  • Hu et al., Squeeze-and-Excitation Networks (2017), arXiv:1709.01507: the channel gate that Gated Residual’s read resembles.
  • Yang and Gao, mHC-lite (2026), arXiv:2601.05732: mixes without Sinkhorn, and per-token measurements of its error.
↑