Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionmulti-head latent attention

single mention · model architecture

Ragged Multi-head Latent Attention

Ragged multi-head latent attention is an attention scheme that mixes layer types: only one in four layers uses full attention while the rest use sliding-window attention.1 It is used to reduce the KV-cache footprint of the model.2

Trade-off It cuts KV-cache size substantially with negligible pretraining impact, but models using it showed poor post-SFT performance on long-context agentic evaluations.3,4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. One in every four layers uses full attention, the remaining layers use sliding-window attention (SW A) [18–20] with a window size of 512 tokens — technical report
  2. we used multiple optimizations to reduce the KV-cache footprint of ML4, notably: • Ragged multi-head latent attention (MLA — technical report
  3. reduced the KV-cache size by a factor of 16 compared to a full-attention baseline — technical report
  4. the model’s post-SFT performance was lower than expected on agentic evaluations, especially those that require long-context reasoning — technical report

Also called Ragged multi-head latent attention (MLA).

Filed under model architecture › token mixer › softmax attention › multi-head latent attention