single mention · model architecture
Three-block causal attention mask
Partitions attention into three causal blocks while allowing bidirectional attention within each block.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See softmax attention for the methods filed beside it.
Also called blockwise causal attention mask.
Filed under model architecture › token mixer › softmax attention