single mention · model architecture
MHA and MQA modes of MLA
An implementation that uses MHA mode for training and prefilling and MQA mode for decoding.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See multi-head latent attention for the methods filed beside it.
Filed under model architecture › token mixer › softmax attention › multi-head latent attention