Open Models Almanac
Methodsmodel architecturetoken mixerlinear attention & state spacegated delta network

specific method · filed under model architecture

Kimi Delta Attention

A linear-attention method that extends the delta-rule recurrence with a channel-wise forget gate.

The sources on this page name it without explaining how it works. The line above is the extraction's one-line gloss. See gated delta network for the methods filed beside it.

Also called KDA linear attention layers, Kimi Delta Attention (KDA).

sources
6
models
2
labs adopt it
2
strongest
core

Documented in

Moonshot AI
Z.ai (Zhipu AI)

Evidence

6 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 5

Kimi K3

Moonshot AIadopted5 sources

Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), two architectural updates designed to improve how information flows across sequence length and model depth.

coremodel architecturein Kimi K3Moonshot AI
4 more from this model

KDA extends the delta-rule recurrence [105, 138] with a channel-wise forget gate [63].

coremodel architecturein Kimi K3Moonshot AI

Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes)

coremodel architecturein Kimi K3Moonshot AI

Kimi K3 is built on Kimi Delta Attention (KDA)

coremodel architecturein Kimi K3Moonshot AI

Kimi Delta Attention, a hybrid linear attention scheme

usedmodel architecturein Kimi K3Moonshot AI

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑