single mention · model architecture
Input-dependent full-rank output gate
An input-dependent full-rank output gate scales a gated linear-attention layer's recurrent output per token before the output projection.1 A full-rank data-dependent projection produces gate values that multiply the RMSNorm-ed recurrent output before the output projection.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
changes KDA’s output gate from the low-rank parameterization used by Kimi Linear [63] to an input-dependent full-rank projection
— k3 tech reportyt =W o[Sigmoid(Wgxt)⊙RMSNorm( ˜ot
— k3 tech report
Also called Input-dependent full-rank output gating, input-dependent full-rank projection.
Filed under model architecture › token mixer › linear attention & state space › gated delta network