single mention · model architecture
Widened residual stream
A widened residual stream replaces the single residual vector with several parallel branches that the network carries through its blocks.1 Each block reads its input as a learnable weighted sum of the branches and writes its output back to a single branch chosen round-robin by depth.2,3
Trade-off The extra compute is negligible (only learnable scalars), but memory traffic grows with the number of branches.4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
replace the single residual vector with several parallel branches
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilityreads its input as a weighted sum of the branches
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitywritten back to a single branch, chosen round-robin by depth
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilityno matrix multiplication, so its compute cost is negligible; the extra cost is the memory traffic
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Also called widening the residual stream.
Filed under model architecture › normalization & residual