Open Models Almanac
Methodsmodel architecturetoken mixerlinear attention & state spacegated delta network

filed under model architecture

KDA

A hybrid linear-attention method described as preserving full expressiveness in critical layers; the evidence does not establish whether this use of KDA is identical to Kimi Delta Attention.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See gated delta network for the methods filed beside it.

Also called Kimi Distributed Attention (KDA).

sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Moonshot AI

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
core 2

Kimi K3

Moonshot AIadopted2 sources

A hybrid linear attention mechanism that replaces standard quadratic attention in a subset of layers. KDA maintains full expressiveness for critical layers

coremodel architecturein Kimi K3Moonshot AI
1 more from this model

Its architecture uses KDA and Attention Residuals for computational efficiency.

coremodel architecturein Kimi K3Moonshot AI
↑