Qwen3.8-Flash-Next
Alibaba (Qwen)the write gate is a single scalar per branch
1 more from this model
The block output is written to every branch through one data-dependent scalar per branch
implementation detail · filed under model architecture
A per-branch scalar residual write gate scales what each residual branch writes into the residual stream using one data-dependent scalar per branch.1 It complements an element-wise data-dependent read gate, being deliberately coarse where per-element parameters would add inference cost.2
Trade-off It provides finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low.3
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
the write gate is a single scalar per branch— Qwen3.8-Flash-Next — Architecture, Evals & Inference Performance
That asymmetry is the design: expressive where it is cheap to be expressive, and deliberately coarse where per-element parameters would add inference cost for little return— Qwen3.8-Flash-Next — Architecture, Evals & Inference Performance
finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low— Qwen3.8-Flash-Next — Architecture, Evals & Inference Performance
Also called write gate.
2 quotes from 2 sources, one card per model, each opening with its strongest.
the write gate is a single scalar per branch
The block output is written to every branch through one data-dependent scalar per branch