Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attention

single mention · model architecture

Three-block causal attention mask

Partitions attention into three causal blocks while allowing bidirectional attention within each block.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See softmax attention for the methods filed beside it.

Also called blockwise causal attention mask.

Filed under model architecture › token mixer › softmax attention