Kimi K3
Moonshot AIKimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), two architectural updates designed to improve how information flows across sequence length and model depth.
4 more from this model
KDA extends the delta-rule recurrence [105, 138] with a channel-wise forget gate [63].
Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes)
Kimi K3 is built on Kimi Delta Attention (KDA)
Kimi Delta Attention, a hybrid linear attention scheme