Model techniques map
Techniquesmodel architecturetoken mixerlinear attention & state spacegated delta network

specific method · filed under model architecture

Kimi Delta Attention

A delta-rule linear-attention method that extends the recurrence with a channel-wise forget gate.

Also called KDA linear attention layers, Kimi Delta Attention (KDA).

sources
6
models
2
labs adopt it
2
strongest
core

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 1core 5

Documented in

GLM-5.3-FlashKimi K3

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Evidence

6 spans quoted from the sources, strongest treatment first.

Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes)

coremodel architecturein Kimi K3Moonshot AI
MoonshotAI/Kimi-K3·Moonshot AI

Kimi K3 is built on Kimi Delta Attention (KDA)

coremodel architecturein Kimi K3Moonshot AI
Kimi K3: The open-weights escalation·Interconnects (Nathan Lambert)·2026-07-20

KDA extends the delta-rule recurrence [105, 138] with a channel-wise forget gate [63].

coremodel architecturein Kimi K3Moonshot AI
k3 tech report·Kimi Team

Its 45-layer language model combines KDA linear-attention layers with NoPE sparse MLA layers

coremodel architecturein GLM-5.3-FlashZ.ai
zai-org/GLM-5.3-Flash — 321B / 18B active · MOE · 1024K ctx·vLLM·2026-09-05

Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), two architectural updates designed to improve how information flows across sequence length and model depth.

coremodel architecturein Kimi K3Moonshot AI
Kimi K3 Tech Blog: Open Frontier Intelligence·Moonshot AI·2026-07-16

Kimi Delta Attention, a hybrid linear attention scheme

usedmodel architecturein Kimi K3Moonshot AI
China's 2.8-trillion-parameter Kimi K3 beats Claude Fable 5 in Frontend Code Arena benchmark— Moonshot AI delivers largest open-weight AI…·Tom's Hardware·2026-07-17

Filed alongside

Other methods under model architecture :: token mixer :: linear attention & state space :: gated delta network.

Gated DeltaNetGated DeltaNet–sparse MoE hybridKDAGated DeltaNet with bounded sigmoid output gateGated DeltaNet with reduced KV-head configurationHybrid linear attentionKimi Delta Attention and Attention Residuals architectureKimi Delta Attention with input-dependent full-rank output gateKimi Delta Attention with lower-bounded log-decaySimpleGDN