Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attention

general family · filed under model architecture

Causal attention

Restricts attention so tokens can attend only to preceding tokens, with the mask enforcing that restriction in the cited implementation.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See softmax attention for the methods filed beside it.

Also called causal attention mask.

sources
2
models
2
labs adopt it
2
strongest
used

Documented in

Hugging Face

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2
↑