Open Models Almanac
Methodsinference & servinginference kernel

specific method · filed under inference & serving

FlashAttention

FlashAttention computes exact attention in one kernel by splitting queries, keys and values into blocks that fit in shared memory and keeping a running maximum, sum and output per query row (the online softmax), so the full matrix of scores is never written to HBM. Its versions 2 to 4 rewrite the same algorithm for each GPU generation's units.

Written for the explainer Kernels and graph compilation: where the cost model meets the GPU.

sources
2
models
2
labs adopt it
2
strongest
used

Documented in

MiniMax
OpenVLA project (Stanford et al.)

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
optional 1used 1

MiniMax-M3

MiniMaxadopted1 source

MSA's kernel, described in section 4, reuses the FlashAttention algorithmic skeleton

usedsoftware implementationin MSA inference kernelMiniMax
↑