Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionattention sink

single mention · model architecture

Attention sink

An attention sink is a token, usually an early one, on which heads put attention they have no use for. Softmax weights must add up to one, so a head with nothing to pass on needs somewhere to put them.

Written for the explainer Attention sinks and output gates.

Filed under model architecture › token mixer › softmax attention › attention sink