Open Models Almanac
Methodsmodel architecturenormalization & residual

single mention · model architecture

Widened residual stream

A widened residual stream replaces the single residual vector with several parallel branches that the network carries through its blocks.1 Each block reads its input as a learnable weighted sum of the branches and writes its output back to a single branch chosen round-robin by depth.2,3

Trade-off The extra compute is negligible (only learnable scalars), but memory traffic grows with the number of branches.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. replace the single residual vector with several parallel branches — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  2. reads its input as a weighted sum of the branches — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  3. written back to a single branch, chosen round-robin by depth — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  4. no matrix multiplication, so its compute cost is negligible; the extra cost is the memory traffic — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

Also called widening the residual stream.

Filed under model architecture › normalization & residual