MiniMax-M3
MiniMaxMSA's kernel, described in section 4, reuses the FlashAttention algorithmic skeleton
specific method · filed under inference & serving
FlashAttention computes exact attention in one kernel by splitting queries, keys and values into blocks that fit in shared memory and keeping a running maximum, sum and output per query row (the online softmax), so the full matrix of scores is never written to HBM. Its versions 2 to 4 rewrite the same algorithm for each GPU generation's units.
Written for the explainer Kernels and graph compilation: where the cost model meets the GPU.
2 quotes from 2 sources, one card per model, each opening with its strongest.
MSA's kernel, described in section 4, reuses the FlashAttention algorithmic skeleton
supports modern techniques for large transformer model training such as automatic mixed precision (AMP, PyTorch [75]), FlashAttention [76]