single mention · model architecture
Ragged Multi-head Latent Attention
Ragged multi-head latent attention is an attention scheme that mixes layer types: only one in four layers uses full attention while the rest use sliding-window attention.1 It is used to reduce the KV-cache footprint of the model.2
Trade-off It cuts KV-cache size substantially with negligible pretraining impact, but models using it showed poor post-SFT performance on long-context agentic evaluations.3,4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
One in every four layers uses full attention, the remaining layers use sliding-window attention (SW A) [18–20] with a window size of 512 tokens
— technical reportwe used multiple optimizations to reduce the KV-cache footprint of ML4, notably: • Ragged multi-head latent attention (MLA
— technical reportreduced the KV-cache size by a factor of 16 compared to a full-attention baseline
— technical reportthe model’s post-SFT performance was lower than expected on agentic evaluations, especially those that require long-context reasoning
— technical report
Also called Ragged multi-head latent attention (MLA).
Filed under model architecture › token mixer › softmax attention › multi-head latent attention