Kimi K3
Moonshot AIMLA caches ct and reconstructs the content keys and values through learned up-projections during attention computation.
2 more from this model
Gated MLA: Gated Multi-head Latent Attention for memory-efficient KV-cache management
specific method · filed under model architecture
Gated Multi-head Latent Attention is a variant of Multi-head Latent Attention that compresses each token's key–value representation into a low-dimensional latent vector, caching that instead of full keys and values.1,2 Keys and values are reconstructed through learned up-projections during attention, and an input-dependent, channel-wise full-rank output gate is applied to the MLA output.3,4
Trade-off The factorization reduces the KV-cache footprint while retaining global token-to-token attention.5
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
compresses the key–value representation of each token into a low-dimensional latent vector— k3 tech report
Instead of caching full head-specific keys and values, MLA caches— k3 tech report
reconstructs the content keys and values through learned up-projections during attention computation— k3 tech report
augments MLA with an input-dependent, channel-wise full-rank output gate— k3 tech report
This factorization reduces the KV-cache footprint while retaining global token-to-token attention— k3 tech report
Also called Gated MLA.
3 quotes from 3 sources, one card per model, each opening with its strongest.
MLA caches ct and reconstructs the content keys and values through learned up-projections during attention computation.
Gated MLA: Gated Multi-head Latent Attention for memory-efficient KV-cache management
Picked by hand, not extracted: where to read more, not evidence for anything on this page.