general family · filed under model architecture
Kimi Delta Attention and Attention Residuals architecture
An architecture pairing Kimi Delta Attention with Attention Residuals; the evidence names the combination but does not describe a distinct combined mechanism.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
core 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes)
coremodel architecturein Kimi K3Moonshot AI
Filed alongside
Other methods under model architecture :: token mixer :: linear attention & state space :: gated delta network.
Gated DeltaNetKimi Delta AttentionGated DeltaNet–sparse MoE hybridKDAGated DeltaNet with bounded sigmoid output gateGated DeltaNet with reduced KV-head configurationHybrid linear attentionKimi Delta Attention with input-dependent full-rank output gateKimi Delta Attention with lower-bounded log-decaySimpleGDN