single source · model architecture
Block Attention Residuals
Block Attention Residuals partitions a model's layers into blocks and reduces each block's layer outputs to a single representation by summation.1,2 Attention layers then attend over only the N block-level representations instead of all layer outputs, so memory and communication overhead drop from O(Ld) to O(Nd).3,4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
we partition the L layers into N blocks of S=L/N layers each
— k3 tech reportlayer outputs are reduced to a single representation by summation
— k3 tech reportfull attention is applied over only the N block-level representations
— k3 tech reportmemory and communication overhead drop from O(Ld) to O(N d
— k3 tech report
Also called block attention residual, Block AttnRes.
Filed under model architecture › normalization & residual