Open Models Almanac
Methodsmodel architecturenormalization & residual

single source · model architecture

Block Attention Residuals

Block Attention Residuals partitions a model's layers into blocks and reduces each block's layer outputs to a single representation by summation.1,2 Attention layers then attend over only the N block-level representations instead of all layer outputs, so memory and communication overhead drop from O(Ld) to O(Nd).3,4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we partition the L layers into N blocks of S=L/N layers each — k3 tech report
  2. layer outputs are reduced to a single representation by summation — k3 tech report
  3. full attention is applied over only the N block-level representations — k3 tech report
  4. memory and communication overhead drop from O(Ld) to O(N d — k3 tech report

Also called block attention residual, Block AttnRes.

Filed under model architecture › normalization & residual