Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionmulti-head latent attention

specific method · filed under model architecture

Gated Multi-head Latent Attention

Gated Multi-head Latent Attention is a variant of Multi-head Latent Attention that compresses each token's key–value representation into a low-dimensional latent vector, caching that instead of full keys and values.1,2 Keys and values are reconstructed through learned up-projections during attention, and an input-dependent, channel-wise full-rank output gate is applied to the MLA output.3,4

Trade-off The factorization reduces the KV-cache footprint while retaining global token-to-token attention.5

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. compresses the key–value representation of each token into a low-dimensional latent vector — k3 tech report
  2. Instead of caching full head-specific keys and values, MLA caches — k3 tech report
  3. reconstructs the content keys and values through learned up-projections during attention computation — k3 tech report
  4. augments MLA with an input-dependent, channel-wise full-rank output gate — k3 tech report
  5. This factorization reduces the KV-cache footprint while retaining global token-to-token attention — k3 tech report

Also called Gated MLA.

sources
3
model
1
lab adopt it
1
strongest
core

Documented in

Moonshot AI

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 1

Kimi K3

Moonshot AIadopted3 sources

MLA caches ct and reconstructs the content keys and values through learned up-projections during attention computation.

coremodel architecturein Kimi K3Moonshot AI
2 more from this model

Gated MLA: Gated Multi-head Latent Attention for memory-efficient KV-cache management

usedmodel architecturein Kimi K3Moonshot AI

69 KDA + 24 Gated MLA

usedmodel architecturein Kimi K3Moonshot AI

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑