Open Models Almanac
Methodsmodel architecturenormalization & residual

specific method · filed under model architecture

Hyper-Connections

Hyper-Connections (HC) replaces the single residual vector with several parallel branches that widen the residual stream.1 It uses learnable read, write, and mixing operators, predicted from the residual state, to form block inputs, distribute block outputs over branches, and exchange information between branches.2,3

Trade-off HC adds parameters per block with negligible compute cost, the main extra cost being memory traffic from carrying multiple branches.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. replace the single residual vector with several parallel branches — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  2. A read operator forms the block input, a write operator distributes the block output over branches, and a mixing operator exchanges information between branches — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  3. The three operators are predicted from the residual state — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  4. This adds parameters per block and no matrix multiplication, so its compute cost is negligible; the extra cost is the memory traffic of carrying branches instead of one — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

Also called Hyper-Connections (HC).

sources
2
models
2
lab adopt it
1
strongest
core

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1core 1
↑