Open Models Almanac
Methodsmodel architecturenormalization & residual

implementation detail · filed under model architecture

Per-branch scalar residual write gate

A per-branch scalar residual write gate scales what each residual branch writes into the residual stream using one data-dependent scalar per branch.1 It complements an element-wise data-dependent read gate, being deliberately coarse where per-element parameters would add inference cost.2

Trade-off It provides finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. the write gate is a single scalar per branch — Qwen3.8-Flash-Next — Architecture, Evals & Inference Performance
  2. That asymmetry is the design: expressive where it is cheap to be expressive, and deliberately coarse where per-element parameters would add inference cost for little return — Qwen3.8-Flash-Next — Architecture, Evals & Inference Performance
  3. finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low — Qwen3.8-Flash-Next — Architecture, Evals & Inference Performance

Also called write gate.

sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Alibaba (Qwen)

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
core 2
↑