Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionmulti-head latent attention

single mention · model architecture

MHA and MQA modes of MLA

An implementation that uses MHA mode for training and prefilling and MQA mode for decoding.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See multi-head latent attention for the methods filed beside it.

Filed under model architecture › token mixer › softmax attention › multi-head latent attention