Open Models Almanac
Methodsmodel architecturetoken mixerlinear attention & state spacegated delta network

single mention · model architecture

Input-dependent full-rank output gate

An input-dependent full-rank output gate scales a gated linear-attention layer's recurrent output per token before the output projection.1 A full-rank data-dependent projection produces gate values that multiply the RMSNorm-ed recurrent output before the output projection.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. changes KDA’s output gate from the low-rank parameterization used by Kimi Linear [63] to an input-dependent full-rank projection — k3 tech report
  2. yt =W o[Sigmoid(Wgxt)⊙RMSNorm( ˜ot — k3 tech report

Also called Input-dependent full-rank output gating, input-dependent full-rank projection.

Filed under model architecture › token mixer › linear attention & state space › gated delta network