Kimi K3
Moonshot AIAttention Residuals (AttnRes) [57] allows each layer to selectively attend to representations from all preceding layers.
6 more from this model
AttnRes selectively retrieves representations across depth rather than accumulating them uniformly.
Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes)
A drop-in replacement for standard residual connections. ... AttnRes allows each layer to selectively retrieve representations from arbitrary earlier layers.
Its architecture uses KDA and Attention Residuals for computational efficiency.
Attention Residuals, which change how information moves between layers
and Attention Residuals (AttnRes)