specific method · filed under model architecture
Gated Residual
A residual method that widens the stream into parallel branches and uses dynamic gates to control information read from and written to those branches.
Also called Gated Residual (GR), Gated residual connections.
- sources
- 6
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
8 spans quoted from the sources, strongest treatment first.
The original single residual stream is expanded into 4 parallel branches, allowing the model to decide dynamically, based on the current content, how much information to read from each branch and how much to write back.
The gated residual "modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and per-branch scalar write gate"
We describe the concrete design of GR below.
the gate decides how that capacity is spent, while also supplying the rescaling that keeps training stable
GR supplies a rescaling that markedly improves training stability
four residual branches dynamically control cross-layer reads and writes.
Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate.
Gated Residual (GR) widens the residual stream into 4 branches and controls reads and writes with a dynamic gate
Filed alongside
Other methods under model architecture :: normalization & residual.