A decoder layer of DeepSeek-V4-Flash reads its input from four vectors per token, where an ordinary transformer layer reads one. Its configuration says hc_mult: 4 and hidden_size: 4096, so between any two sublayers each token carries 16,384 numbers, 32,768 bytes in BF16 instead of 8,192. How the four vectors are combined is learned: before each sublayer one small matrix product computes 24 coefficients per token (4 weights for reading, 4 for writing and 16 for mixing the vectors among themselves). Those products hold 0.28% of the parameters a token uses in each layer (computed below), and DeepSeek reports that the arrangement costs 6.7% of training time by its measure.
The thing changed is the residual connection, , which deep networks have used since ResNet. It makes each layer’s input the plain sum of the embedding and every earlier output, which the Kimi K3 report calls “a bottleneck reminiscent of RNNs over time”. In 2026 five labs in this site’s map shipped something else in at least one model: DeepSeek and Z.ai use manifold-constrained hyper-connections (mHC), Tencent a variant without the mixing, Qwen its Gated Residual, and Moonshot attention over depth. Of the 35 language models whose modeling code this site has read, 11 have one of these; the other 24 keep a single stream.
These designs lower the loss in their authors’ experiments, and most add under 1% to the arithmetic. They cost in two other places. The wider state must be read and written at every sublayer, which takes time when thousands of tokens are processed at once, as in training and in reading a long prompt. And a learned weight on the path that carries the signal through the depth can compound across layers.
What the residual stream is
The primer 00 has the decoder block in full. A block reads a vector of width for each token, runs a sublayer (attention or a feed-forward network) on a normalized copy, and adds the result back:
The vector is the residual stream. R1 covers the norm; this article covers the addition. A model with decoder layers has sublayers, one attention and one feed-forward per layer, and we index them by . We write for the width and for the number of parallel streams, and let include its norm.
Unrolling from sublayer to any later sublayer gives (He et al. 2016)
The signal from sublayer reaches with weight exactly one, the identity mapping, and the gradient at contains the gradient at unchanged, so it is “unlikely to be canceled out” however deep the stack.
He and colleagues argued that the weight should stay at one. With a scaled shortcut, , the unrolled sum begins with : over the 86 sublayers of DeepSeek-V4-Flash, gives 2.35 and gives 0.42 (computed). On CIFAR-10, scaling the shortcut and branch of a 110-layer ResNet by 0.5 raised the error from 6.61% to 12.35%.
In a pre-norm transformer a constant scalar on the shortcut does less harm than this suggests, because every sublayer normalizes its input and RMSNorm ignores overall scale. Rescaling the stream by shows the network computes the same function as a plain residual whose branch is weighted by : at , from 0.99 for the first sublayer to 0.42 for the last (our derivation, computed). A matrix on the shortcut can’t be absorbed this way: the norm removes only one common scale, and a product of matrices, even of one matrix repeated, also turns directions. (A fixed invertible mix can be moved into the read and write weights instead, which then carry the compounding.) That is the case the widened designs have to handle.
Qwen’s report gives the reason to change the residual at all, in the pre-norm network: “every block reads the same stream, so a feature written early must compete with everything written after it.” One way to see the options is to write the input of every sublayer as a weighted sum of earlier outputs,
with the embedding and the output of sublayer . For the plain residual every weight is 1, a lower-triangular matrix of ones, whose blocks below the diagonal all have rank one. The Attention Residuals paper classifies the designs below by the structure they give (its §6.2), and we follow it.
Where the bytes go
A GPU computes in small, fast on-chip memory and stores tensors in large, slower main memory. A kernel is one program launched on the GPU; each kernel reads its inputs from main memory and writes its outputs back. Memory traffic is the bytes moved that way, and kernel fusion puts several operations into one kernel so the intermediate results stay on chip. A kernel that reads the whole residual state makes one pass over it.
Whether those bytes cost time depends on how many tokens run at once. Generating text (decode) produces one token per sequence per step; every step reads the weights the batch uses (in a mixture-of-experts model, the shared weights and the experts the tokens chose), and with few sequences that read takes most of the time. Reading a prompt (prefill) and training process thousands of tokens per step, so each weight byte is used thousands of times and the arithmetic dominates. Activation bytes behave the other way: they grow with the number of tokens. In DeepSeek-V4-Flash one layer reads about 209 MB of weights per decode step (experts at FP4, the rest at FP8), while its four-stream mHC layer, with the best fused kernel a V4 model can use (described below), adds 163,840 bytes of stream traffic per token over a plain residual. At one token that is 0.08% of the step’s bytes. The weight bytes stop growing once every expert is in use, which with 6 of 256 experts per token happens within a few hundred tokens, at about 3.35 GB per layer; the stream’s bytes keep growing with the tokens and reach that at about 20,000 tokens per step (computed, BF16 stream). So the stream’s bytes cost time in prefill, training and large-batch serving, and small-batch decode pays mostly in extra kernels run one after another, and in the residual mechanism’s own weights.
Pipeline parallelism splits the layers into consecutive groups (stages), one per group of GPUs, and sends each token’s residual state from one stage to the next; a wider stream sends more. Activation checkpointing, also called recomputation, keeps fewer intermediate results for the backward pass and recomputes the rest. Tensor parallelism splits each weight matrix across GPUs; the residual state is then usually copied on every GPU, and the sublayer’s output is summed across them (an all-reduce) before it is added back.
The wider state lives only while a token passes through the layers. The KV cache, the per-token memory attention keeps across steps, is unchanged by every design here, because attention and the experts still see one -wide input.
Widening with fixed weights
The simplest change keeps the layers and gives the stream more room. Replace the single vector by vectors per token, the streams or branches, start them as copies of the embedding, let each sublayer read a weighted sum of them, and write its output back. At the end the streams are combined into one vector for the output head.
Google’s AltUp (Baykal et al. 2023) did this for T5 models. Qwen tested how much width alone buys with a stripped-down version in a pre-norm decoder: each block holds learned scalars , reads from the branches , and adds its output to branch , round-robin by depth. That is extra parameters per block and no matrix product. On a 25B-parameter mixture-of-experts model with 3B active, trained on 400B tokens, it “lowers the training loss … by roughly 0.01” (Qwen3.8-Next report, §2.2).
In the language of , a sublayer’s weight on an earlier output now depends on which branch that output went to, so the weights repeat with period along the sources, and every block of below the diagonal has rank at most . Width is rank: streams keep up to independent combinations of the past instead of one sum.
- fixed weight
- learned, the same for every token
- computed from the token
Constraint: none: both weights are fixed at 1.
Width costs memory traffic, not arithmetic; in Qwen’s words, “the extra cost is the memory traffic of carrying branches instead of one” ( is our ). With a fixed read one pass suffices, reading the streams and the sublayer output and writing one stream and the next input: about 7d per sublayer at against 4d for a plain residual (computed, in the accounting of the traffic chart below). The form of the weights matters too: on a 1B-parameter OLMo model trained on 500B tokens, Zhu et al. found AltUp with two blocks ended slightly worse than the baseline (validation loss 2.827 against 2.811), while static hyper-connections, the next design with its coefficients held fixed, ended better (2.799 at two streams, 2.791 at four).
Hyper-connections: learning the read, the write and the mix
Hyper-connections (HC; Zhu et al. 2024, ByteDance) make all three operations learned and let them depend on the token. Stack the streams of one token as the rows of . A sublayer then computes
where is the read (the sublayer’s -wide input as a weighted sum of streams), the write (how much of the output each stream receives) and the mix. The sublayer still sees one vector of width , so attention and the experts are unchanged.
Each map is a learned static part plus a token-dependent part: a small linear map of the normalized streams, passed through and scaled by a learned factor that starts at 0.01. At initialization the streams are copies of the embedding, the read takes one stream, the write adds the output to all of them and the mix is the identity. Every stream then receives the same updates, so each one equals the pre-norm transformer’s residual until training moves the coefficients.
On OLMoE-1B-7B (1.3B active of 7B, 500B tokens) four streams lowered the training loss by about 0.027 and the C4 validation loss by 0.028, and the authors report reaching the baseline’s loss 1.8 times sooner. On a 7B dense model the validation loss fell from 2.322 to 2.304, and the baseline showed “frequent spikes” while the HC model showed none. Arithmetic rose by 0.15 to 0.21%.
Activation memory in training rose more: by 26% for a 1B model and 28% for the 7B model at four streams, measured on eight GPUs, and by 9.7% for the mixture-of-experts OLMoE. Static HC cost the same 26% as the token-dependent version, so the cause is the four-times-wider state each sublayer keeps for the backward pass, not the coefficients. HC reports no wall-clock time and no memory-traffic or pipeline numbers.
mHC: bounding the product
Unroll HC’s update and the term that carries forward to sublayer is an ordered product of mixes, . If the factors drift from the identity, the product’s gain along some direction can grow or shrink geometrically in the number of factors; that is what compounding means here. Xie et al. (DeepSeek) measured it on a 27B model, with the largest absolute row sum of the product as the forward gain and the largest column sum as the backward gain, and found “peaks of 3000” for HC, with a loss surge around step 12,000 that tracked the gradient norm.
mHC constrains the mix to doubly stochastic matrices: nonnegative entries, every row and every column summing to one. Each such matrix is a weighted average of permutation matrices (the Birkhoff–von Neumann theorem), so the mix can only shuffle and average the streams. A product of two such matrices is again one, since and likewise for columns. Each output stream is a weighted average of the inputs, so the forward signal can’t grow; the backward pass multiplies by the transpose, whose rows are the columns, so neither can the gradient. Together these make the spectral norm exactly 1 (at most 1 from the two sums, and the all-ones vector attains it). With the only such matrix is the number 1, the plain residual.
What this does to the streams can be said exactly (our derivation). Because the columns sum to one, the mean of the streams, , follows
a plain residual stream with a bounded write weight. The differences between the streams are multiplied by the mixes, which can only shrink them, and refilled by unequal writes: they act as fading memories of recent outputs. How fast they fade depends on how far the mixes are from permutations. A mix whose entries are all at least shrinks the spread between the streams by a factor of at least per sublayer (our derivation), so with every entry at 0.05 or more the carried part is within 5% of copies after about 14 sublayers; mixes close to permutations keep the streams apart.
- unconstrained I + εG (band: 20 draws)
- softmax only, no Sinkhorn
- Sinkhorn, as in DeepSeek-V4's code (mHC)
Toy, at depth 86: unconstrained I + 0.10·G 5.26 (20 draws: 3.40 to 21.3), softmax only, no Sinkhorn 1.64, Sinkhorn with 20 iterations 1.02. Gain = the larger of the largest absolute row sum and column sum of the product.
How different the 4 streams can still be: second singular value of the Sinkhorn product σ2 = 9.0 × 10−12 at depth 86 (the largest is 1; σ2 near 0 means the product has averaged the four streams into one).
To land in the set, mHC exponentiates the raw coefficients and alternately rescales rows and columns to sum to one, the Sinkhorn–Knopp iteration (1967). For a strictly positive matrix this converges to the unique doubly stochastic matrix of the form , with diagonal (an entropic projection), slowest for mixes close to a permutation. DeepSeek stops after 20 (hc_sinkhorn_iters: 20), and the gradient flows through all 20 steps, so the model trains with the same truncated matrix it runs with.
Which sums come out exact depends on the last step. The Transformers port of DeepSeek-V4 (DeepseekV4HyperConnection) starts from a row softmax, ends on a column normalization and applies the transpose, so in the convention of the equation above every row of the mix sums to one, exactly in FP32, and the forward gain of any product is 1; the port casts the mix to BF16 before applying it, which moves a row sum by up to about 0.3% (computed). The columns sum to one only up to the truncation. That error is the backward gain the mHC paper measures at “approximately 1.6” at most, against “nearly 3000” for HC, and it is also the error in the conservation of the stream mean. These are token averages; collected token by token in their own re-implementation on models up to about 0.36B parameters, Yang and Gao (mHC-lite, 2026) found many column sums far from 1.
The read and the write get a lighter bound: a sigmoid for the read, which keeps its weights positive (the sublayer’s norm removes their overall scale), and twice a sigmoid for the write, between 0 and 2. Both prevent “signal cancellation” from mixed signs.
On the 27B model mHC ended with a loss 0.021 below the baseline’s and scored above HC on seven of eight benchmarks: BBH rose from 43.8 (baseline) and 48.9 (HC) to 51.0, and MATH was the exception at 26.4 for HC and 26.0 for mHC. HC still gained on every benchmark despite the spike; the paper’s case is that mHC keeps the gain without an instability that grows with scale. A forward gain that scales every stream alike is invisible to the normalized sublayers; one that grows along some directions changes what they read, and it shows up as early writes that drown later ones and as backward gain.
What mHC costs
In DeepSeek-V4-Flash one mHC module has a projection of weights, plus 27 biases and gains: 393,243 parameters. There are two per layer, so 33.9M for the model including the output collapse (computed from the code shapes). That is 0.28% of the 284M parameters a token uses per layer, and the module’s arithmetic is 0.36% of the layer’s. The code keeps these weights in FP32 while the experts are FP4 and the rest FP8, so in decode they are 135.5 MB per step, 1.5% of the weight bytes at batch 1 (computed). In prefill a product with only 24 outputs uses the GPU’s arithmetic poorly, so its share of time exceeds its share of arithmetic.
The mHC paper counts what one layer of HC reads and writes per token outside the sublayer, unfused: reads and writes plus a few coefficients, against and for a plain residual, 34d against 3d at , about 11 times as much. Pipeline parallelism sends times the data between stages. DeepSeek’s V4 report lists three counter-measures: fused kernels for training and inference; recomputing most mHC intermediates in the backward pass instead of storing them; and an adjusted DualPipe, DeepSeek’s pipeline schedule that runs the transfers between stages while the GPUs compute. With all three, mHC takes “6.7% of the overlapped 1F1B pipeline stage” in wall time during training (1F1B is the schedule in which each stage alternates one forward and one backward micro-batch).
Under tensor parallelism the four streams are copied on every GPU of the group, each GPU repeats the coefficient product, Sinkhorn and the update, and the all-reduce of the sublayer output must finish before the write. The vLLM code for V4.1 fuses that all-reduce into the mHC kernel for one configuration (four GPUs, up to 16 tokens).
Single-pass mHC: one traversal per boundary
DeepSeek changed the model in V4.1-Flash to save one more pass. The V4.1 report counts the traffic per sublayer boundary in elements of width . The ideal reads the streams and the sublayer output once and writes the new streams and the next sublayer’s input once: , 10d at , the same count that gives 4d for a plain residual at . V4’s training implementation ran three mHC kernels plus the pre-norm and moved , 20d.
The obstacle is a dependency. The next sublayer’s read coefficients are a function of all numbers of the updated streams, through a norm and the projection. A kernel works through the streams one tile at a time, so it can’t apply the coefficients to the first tile before it has seen the last. One pass computes the coefficients and a second reads the stream values again to apply them: , 14d. Single-pass mHC removes the dependency by shifting the read by one step: each sublayer reads with coefficients computed by the previous one, which are known before the pass starts. V4.1 also drops V4’s learned output collapse in favour of the last read. With the shift, the serving kernel Mega-mHC reaches the 10d bound. DeepSeek keeps the older multi-kernel code for pre-training, “since the shift only changes which mixing coefficients each block applies”, so the model is pre-trained with the shift and served with the fused kernel; the report says the shift “incurs negligible performance degradation” without a number. Once fused, the mix itself costs no extra pass; the dependency does.
Other parts of V4.1 work on four streams too: its three draft layers for speculative decoding are full decoder layers with their own mHC, fed the main model’s stream mean, and its Engram module, a hashed n-gram memory, writes into each stream separately.
Tencent’s variant: no mix at all
Hy4-preview keeps four streams and DeepSeek’s form of the read and write, token-dependent, but has no mixing matrix and no Sinkhorn. Its documents call it iHC, “identity Hyper-Connections”; the code docstring says “Independent Hyper-Connection”. Each stream is updated as : the identity path of every stream is the identity, so there is no product to bound. hc_magnitude: 2.0 is the upper bound of the write gate, DeepSeek’s 2 made configurable. Without the part a module has 196,618 parameters instead of the 589,851 an mHC module of the same width would have (computed).
Gated Residual: put the capacity in the read
Qwen ran an ablation in its own framework, on a 25B-A3B model trained on 560B tokens, to find where HC’s gain comes from. All variants used four branches.
| Residual design | Loss | Benchmark average |
|---|---|---|
| Pre-norm, one stream | 1.617 | 50.91 |
| mHC, static coefficients | 1.596 | 52.49 |
| mHC, token-dependent coefficients | 1.594 | 54.47 |
| Gated Residual | 1.590 | 54.66 |
The static widened stream takes 0.021 off the loss; making the coefficients depend on the token takes 0.002 more but adds 1.98 benchmark points. A sigmoid beat . Refining the read from one scalar per branch to one weight per branch and channel helped, while the same refinement of the write “gives almost nothing”. And once the read and write were expressive, “adding the mixing operator brings no significant improvement.”
Gated Residual (GR), in Qwen3.8-Flash-Next (6B active parameters), is built from those findings. With the branch after its own RMSNorm and all four stacked,
where the gate has one entry per branch and channel, and form a bottleneck of rank 320 over the stacked channels (hc_count: 4, hc_lowrank: 320), and is one write scalar per branch, also predicted from . There is no mix, so every branch keeps an identity path and nothing compounds. The read is the GatedNorm of R1 applied to the widened stream, and a squeeze-and-excitation gate (Hu et al. 2017) whose squeeze is over the token’s own branches instead of over image positions, with a reduction ratio of 32. Because it already normalizes, GR replaces the block’s pre-norm.
GR is the one design here with a visible arithmetic cost. A module has 6.6M parameters, almost all of them in the bottleneck, 27 times an mHC module at the same width. With two per layer and a read-only one before the output head, 640.6M parameters run for every token, about 11% of the 6 billion active parameters Qwen reports, and in BF16 they are 1.28 GB read per decode step (computed). In return GR matched Qwen’s own mHC in the ablation above. Qwen’s stress test on a 28-layer 25B-A3B model, with Muon at four times the optimal learning rate, found that “the configuration with the gated residual records zero loss spikes”, and adding GR to the Qwen3.5 structure lowered the loss at 276B tokens of the production configuration by 0.026.
By Qwen’s account, dropping the mix also removes “a full read of the residual state per block”. That holds against an unfused mHC. With a fused write, GR moves the same 14d as two-pass mHC (computed): its gate, like mHC’s read, needs a reduction over all branches before the read can start, the dependency DeepSeek removed with its shift.
Qwen stores the branches in FP8, because bounded gates in GR, in gated attention and in Gated DeltaNet “all bound the magnitude of what is written into the stream”; FP8 “halves the bytes moved for the residual state relative to BF16, with almost no loss in quality.” The report gives no number for the loss and no format or scaling scheme, and the pinned Transformers code keeps the stream in BF16. Qwen also trained a variant, from scratch and from mid-training, in which each block reads only the two branches with the highest gates. Pre-training loss and benchmarks were “almost unaffected”, but “the quality degraded clearly after post-training, so we did not adopt it”: “a case where pre-training metrics alone would have led to the wrong decision”.
Attention residuals: attention over depth
Moonshot took the other route. Instead of more width, attention residuals (AttnRes; Kimi Team 2026) give each sublayer a choice of what to read. In the plain residual sublayer reads the sum of everything before it; in AttnRes it reads a softmax-weighted average of the embedding and each earlier sublayer’s output:
The query is a learned vector, a “pseudo-query”, the same for every token; the weights still depend on the token through the keys. The keys are RMS-normalized so that a sublayer with large outputs can’t take all the weight; the values are not. In terms of , full AttnRes makes the weights dense and token-dependent.
Weighted sums over layers are old: ELMo mixed a network’s layers with learned softmax weights, DenseNet feeds every earlier output to each layer, and DenseFormer (Pagliardini et al. 2024) gives each block static learned weights over all earlier ones. AttnRes adds weights that depend on the token and a query per sublayer. In the paper’s ablation on a 16-layer model with 436M active parameters, DenseFormer’s static weights gained nothing (loss 1.767 against 1.766 for the plain residual), input-independent mixing reached 1.749 and AttnRes 1.737.
Every read is a convex combination, so its size is at most that of its largest source, where the plain sum grows with depth, and an earlier output reaches a later sublayer through a single weight, not a product of weights, so nothing compounds. The price is that no path keeps a weight fixed at one: an early output reaches a later sublayer only as strongly as that sublayer’s softmax weights it, and a source whose weights fall towards zero gets almost no gradient through those reads. The factor is not itself a loss against pre-norm, because the norm after each read rescales the convex average (our derivation). The paper’s measured gradients are more uniform across depth than the baseline’s. With the pseudo-queries initialized to zero the weights are uniform, and since each sublayer normalizes its input, a uniform average gives exactly the pre-norm transformer’s inputs: training starts from the ordinary network (our derivation).
Full AttnRes costs arithmetic per token and memory to keep every output. The paper calls the arithmetic feasible for layers and the K3 report “affordable” at . Memory limits the full form: ordinary training keeps those outputs for the backward pass anyway, but with recomputation and pipeline parallelism every output must stay alive and cross every later stage.
Block attention residuals cut that to the number of blocks. The sublayers are split into blocks; within a block the outputs are summed as in a plain residual, and each sublayer attends over the completed block sums, the embedding, and the running sum of its own block. Memory falls from to , and the paper finds that “N≈8 recovers most of the benefit across model scales”. The block size is a fixed choice; the loss trains only the weights over the sources.
- source held for the read (embedding, a completed block's sum; in full, every earlier output)
- running sum of the current block
- the sublayer reading
Layer 50, attention. Block: sources read: 6 (of 9 at most): the embedding, the sums of layers 0–11, 12–23, 24–35, 36–47, and the running sum of layers 48–49. Held during the forward pass, per token in flight: 6 vectors = 86,016 bytes at BF16. Full: 101 sources (up to 187 at the output), held per token in flight: 101 vectors = 1,447,936 bytes at BF16.
Kimi K3 has 93 layers and attn_res_block_size: 12. The report describes “8 blocks with 12-layer size, giving a partial final block and 9 total blocks when counting the embedding layer”: seven blocks of 12 and a last one of 9 (computed from the config). The code has a pseudo-query per sublayer, so every sublayer after the first and the output head read their own weighted average, 186 reads of 2 to 9 sources each. A token in flight holds at most 9 vectors of 7,168 values, 129,024 bytes at BF16, against 14,336 for a single stream; full AttnRes would hold 187, 20.8 times as many. Over a 128K-token prompt processed in one piece the 9 vectors take 15.75 GiB (computed); the Attention Residuals paper splits them across GPUs and processes the prompt in chunks. AttnRes adds 2.7M parameters and 0.02% of a layer’s arithmetic.
K3 handles these costs in its systems. It wraps the AttnRes computation in activation checkpointing, so the activations saved per layer are “identical to that of the standard residual architecture”, and under pipeline parallelism sends only newly completed blocks between stages (“cache-based pipeline communication”). At inference a two-phase schedule reads the stored block sums once per block in one batched pass, then folds in the running sum sublayer by sublayer with an online-softmax merge: two softmax averages over disjoint sources combine exactly from their outputs and their running maximum and sum of exponentials. In prefill K3 splits the tokens across the tensor-parallel GPUs so each token’s block sums live on one GPU; in decode it fuses the merge into the all-reduce. Its EAGLE-3 draft model reads the outputs of the 1st, 4th and final AttnRes blocks. The Attention Residuals paper reports under 4% training overhead with pipeline parallelism and under 2% inference latency “on typical inference workloads”; the K3 report gives no overhead number.
In the paper’s scaling runs (194M to 528M active parameters), Block AttnRes reached the baseline’s loss with what the authors call a 1.25 times compute advantage, and its mHC baseline, mHC-lite, stayed within 0.006 of it at every size: lower at the three smallest (194M to 296M) and 0.001 higher at the two largest. On the paper’s 48B Kimi Linear model Block AttnRes matched or beat the baseline on every benchmark listed, GPQA-Diamond from 36.9 to 44.4. In Qwen’s runs on a 28-layer model, full AttnRes reached 1.762 without GatedNorm, level with GR’s 1.762 (GR’s read already carries the gate), and 1.758 with it, and Block AttnRes with four sublayers per block reached 1.768 with GatedNorm; K3 sums 24 sublayers per block. Moonshot’s “approximately 2.5×” scaling efficiency of K3 over K2 is credited to four changes together, AttnRes among them.
The paper’s 5.5d per layer for Block AttnRes against 34d for mHC compares with the unfused mHC count above and counts 3d for a plain residual; in the chart’s accounting K3’s Block AttnRes moves about 6.3d against 4d.
Side by side
| Design | Identity path | Depth weights | Constraint | Extra parameters | Elements per boundary, | Reported gain (scope) | Shipped in |
|---|---|---|---|---|---|---|---|
| Plain residual | weight 1 | all ones, rank 1 | none needed | 0 | 4d | baseline | 24 of 35 |
| Width only (simplified AltUp) | identity per stream | periodic, rank ≤ n | none | per block | 7d (computed) | loss −0.01 (Qwen, 25B-A3B, 400B tokens) | none |
| HC | learned product | rank ≤ n | on the token part only | 0.03% (OLMo-1B) | 34d unfused | loss −0.027 (OLMoE-1B-7B) | none |
| mHC | doubly stochastic product | rank ≤ n | sigmoid read, 2σ write, Sinkhorn mix | 33.9M, FP32 (V4-Flash) | 20d → 14d → 10d | loss −0.021, BBH +7.2 (27B) | DeepSeek V4, V4.1; GLM-5.3-Flash |
| iHC | identity per stream | rank ≤ n | sigmoid read, gated write | 30.8M (Hy4-preview) | not published | not published | Hy4-preview |
| Gated Residual | identity per stream | rank ≤ n | sigmoid gates, per-branch norm | 640.6M (Qwen3.8-Flash-Next) | 14d (computed) | loss −0.027, average +3.75 (Qwen, 25B-A3B) | Qwen3.8-Flash-Next |
| Block AttnRes | one convex weight per source | between N and N + S | softmax, normalized keys | 2.7M (Kimi K3) | about 6.3d (computed) | 1.25× compute (194M–528M active) | Kimi K3 |
Where the models in the map sit
From the configs and modeling code in this site’s run, 39 models have a usable code check; 35 are language models, and the four robot models all keep one stream. No code check covers the residual stream yet, so this table is read from the code directly.
| Model | Design | Config fields | Layers × width | From the code |
|---|---|---|---|---|
| DeepSeek-V4-Flash, -0731, -Vision-Exp | mHC | hc_mult: 4, hc_sinkhorn_iters: 20 | 43 × 4096 | two modules per layer; learned weighted collapse at the end |
| DeepSeek-V4-Pro, -0813 | mHC | same | 61 × 7168 | same code as V4-Flash |
| DeepSeek-V4.1-Flash, -1017 | single-pass mHC | same | 40 × 5120 | read shifted by one sublayer; no learned collapse; draft layers carry four streams |
| GLM-5.3-Flash | mHC | same, plus "mhc": true | 45 × 4096 | DeepSeek’s class line for line; plain mean at the end |
| Hy4-preview | iHC | hc_mult: 4, hc_magnitude: 2.0, enable_ihc: true | 78 × 6144 | no mix, no Sinkhorn |
| Qwen3.8-Flash-Next | Gated Residual | hc_count: 4, hc_lowrank: 320 | 48 × 2560 | replaces the pre-norms; a read-only GR before the output head |
| Kimi K3 | Block AttnRes | attn_res_block_size: 12 | 93 × 7168 | 8 stored entries plus a 9-layer partial block |
All 11 come from five labs, and the same labs’ earlier models in the map (DeepSeek-V3.2, GLM-5.2 and GLM-5.3, Hy3, Qwen3.5 and Qwen3.6) use a single stream. No lab that keeps a single stream says it tried a widened or attention-based residual and rejected it. Nobody has published a head-to-head of mHC, Gated Residual and AttnRes at production scale. The closest are Qwen’s runs, which compare its own versions of mHC and AttnRes with GR on 28- and 48-layer models and a 25B-A3B MoE.
Easy to get wrong
- mHC does not keep the signal norm at one. The exact constraint keeps the stream mean a plain residual and shrinks the differences between streams; with DeepSeek’s 20 Sinkhorn steps the forward gain is 1 (exactly in FP32, up to BF16 rounding as applied) and the backward gain is off by the truncation, at most 1.6 in the mHC paper’s token averages, with larger per-token outliers in mHC-lite’s measurements of small models.
- The wider stream does not cost decode time in proportion to its bytes. At small batch the step’s bytes are weights: the stream adds 0.08% in DeepSeek-V4-Flash, while mHC’s FP32 weights add 1.5%; in Qwen3.8-Flash-Next, GR’s weights are about 11% of the active parameters. The stream’s traffic costs time in prefill, training and large batches.
- Single-pass mHC is a model change, not only a faster kernel: each sublayer reads with the previous sublayer’s coefficients. DeepSeek pre-trains with the shift and calls its quality cost “negligible” without a number.
- Kimi K3 does not have 8 blocks of 12 layers. It has 93 layers: seven blocks of 12, a last block of 9, and the embedding as its own source.
- Pre-training loss alone can miss what a residual change does after post-training: Qwen’s top-two-branch read was “almost free in pre-training loss” and was dropped for what it did later.
Check yourself
DeepSeek-V4.1-Flash has hidden_size: 5120 and four streams. How many bytes per token does its residual state take between two sublayers in BF16, and how many activation elements does one boundary move with single-pass mHC, against a plain residual?
The state is values, 40,960 bytes in BF16, against 10,240 for one stream. Single-pass mHC moves elements per boundary, 102,400 bytes; a plain residual moves . The model is the one in the map that was built for the shift; a V4 model would need the two-pass .
The identity path of DeepSeek-V4-Pro (61 layers) passes through how many mixing matrices, and why can’t their product amplify the forward signal?
Two mixes per layer, so 122. Each mix, as applied, has rows that sum to one (exactly, before the cast to BF16) with nonnegative entries, so each output stream is a weighted average of the inputs, and a product of such matrices keeps that property. The columns sum to one only up to Sinkhorn’s truncation, so the backward gain can drift slightly. HC’s free mixes have no such property; their product can grow or shrink geometrically, like He et al.’s .
Why does the fused Mega-mHC kernel need the one-step shift to reach 10d?
Without it, the next sublayer’s read coefficients depend on all values of the updated streams, and the kernel can’t apply them until it has read every tile, so it reads the streams a second time: . With the shift the coefficients come from the previous sublayer and are known before the pass, so one read and one write suffice: .
Kimi K3’s output head reads how many sources, and why is that fewer than the number of layers?
Nine: the 8 stored entries (the embedding and the seven completed sums of layers 0 to 83) and the running sum of layers 84 to 92. Within a block the outputs are summed as in a plain residual, so the model holds one vector per block instead of one per sublayer.
In the map
Methods
- Hyper-Connections 2 sources
- Manifold-Constrained Hyper-Connections 9 sources
- Single-Pass mHC 2 sources
- Identity Hyper-Connections 3 sources
- Gated Residual 6 sources
- Attention Residuals 7 sources
- Block Attention Residuals 1 source
- Full Attention Residuals 1 source
- Simplified AltUp variant 1 source
- FP8 residual-state storage 1 source
Taxonomy nodes
- normalization & residual 36 methods
Further reading
- He et al., Identity Mappings in Deep Residual Networks (2016), arXiv:1603.05027: the unrolled residual and the scaled-shortcut argument.
- Zhu et al., Hyper-Connections (2024), arXiv:2409.19606: the original design, its results and its memory cost.
- Baykal et al., Alternating Updates for Efficient Transformers (2023), arXiv:2301.13310: AltUp’s predict-and-correct widening on T5.
- Xie et al., mHC: Manifold-Constrained Hyper-Connections (2025), arXiv:2512.24880: the gain measurements, the constraint, and the kernel and pipeline design.
- DeepSeek-V4 report (arXiv:2606.19348), §2.2 and §3.4.2, for mHC as shipped and its 6.7% cost; DeepSeek-V4.1 report §2.4.1 for single-pass mHC and the traffic accounting the chart uses.
- Qwen, On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability (arXiv:2608.30320), §2.2 and §3.3: the ablation, Gated Residual and the stability tests.
- Kimi Team, Attention Residuals (2026), arXiv:2603.15031: full and block AttnRes, the depth-weight view (§6.2) and the systems design; the Kimi K3 report §2.2 for the shipped configuration.
- Hu et al., Squeeze-and-Excitation Networks (2017), arXiv:1709.01507: the channel gate that Gated Residual’s read resembles.
- Yang and Gao, mHC-lite (2026), arXiv:2601.05732: mixes without Sinkhorn, and per-token measurements of its error.
