Open Models Almanac
Methodsmodel architecturenormalization & residual

specific method · filed under model architecture

Gated Residual

Gated Residual widens the residual stream into four branches and reads them through a per-channel sigmoid gate computed from all branches through a low-rank bottleneck, then adds each sublayer's output to every branch with one learned scalar per branch. It has no matrix mixing the branches, so each branch keeps a plain identity path.

Written for the explainer Rethinking the residual stream.

Also called Gated Residual (GR), Gated residual connections.

sources
6
model
1
lab adopt it
1
strongest
core

Documented in

Alibaba (Qwen)

Evidence

8 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
core 8

Qwen3.8-Flash-Next

Alibaba (Qwen)adopted6 sources

Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate.

coreunclearin Qwen3.8-Flash-NextQwen Team
7 more from this model

Gated Residual (GR) widens the residual stream into 4 branches and controls reads and writes with a dynamic gate

coremodel architecturein Qwen3.8-Flash-NextQwen Team

the gate decides how that capacity is spent, while also supplying the rescaling that keeps training stable

coreunclearin Qwen3.8-Flash-NextQwen Team

GR supplies a rescaling that markedly improves training stability

coremodel architecturein Qwen3.8-Flash-NextQwen Team

We describe the concrete design of GR below.

coreunclearin Qwen3.8-Flash-NextQwen Team

The gated residual "modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and per-branch scalar write gate"

coremodel architecturein Qwen3.8-Flash-NextQwen Team

The original single residual stream is expanded into 4 parallel branches, allowing the model to decide dynamically, based on the current content, how much information to read from each branch and how much to write back.

coreunclearin Qwen3.8-Flash-NextQwen Team

four residual branches dynamically control cross-layer reads and writes.

coremodel architecturein Qwen3.8-Flash-NextQwen Team
↑