Open Models Almanac
Methodsmodel architecturenormalization & residual

specific method · filed under model architecture

Attention Residuals

Attention residuals replace the fixed sum of a residual stream with a softmax-weighted average of the embedding and earlier layers' outputs, using a learned query per sublayer and normalized keys. The block form sums outputs within blocks of layers and attends over the block sums, so only a few vectors per token are held.

Written for the explainer Rethinking the residual stream.

Also called Attention Residuals (AttnRes), AttnRes.

sources
7
model
1
lab adopt it
1
strongest
core

Documented in

Moonshot AI

Evidence

7 quotes from 7 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 6

Kimi K3

Moonshot AIadopted7 sources

Attention Residuals (AttnRes) [57] allows each layer to selectively attend to representations from all preceding layers.

coremodel architecturein Kimi K3Moonshot AI
6 more from this model

AttnRes selectively retrieves representations across depth rather than accumulating them uniformly.

coremodel architecturein Kimi K3Moonshot AI

Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes)

coremodel architecturein Kimi K3Moonshot AI

A drop-in replacement for standard residual connections. ... AttnRes allows each layer to selectively retrieve representations from arbitrary earlier layers.

coremodel architecturein Kimi K3Moonshot AI

Its architecture uses KDA and Attention Residuals for computational efficiency.

coremodel architecturein Kimi K3Moonshot AI

Attention Residuals, which change how information moves between layers

usedmodel architecturein Kimi K3Moonshot AI

and Attention Residuals (AttnRes)

coremodel architecturein Kimi K3Moonshot AI

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑