single mention · model architecture
Attention sink
An attention sink is a token, usually an early one, on which heads put attention they have no use for. Softmax weights must add up to one, so a head with nothing to pass on needs somewhere to put them.
Written for the explainer Attention sinks and output gates.
Filed under model architecture › token mixer › softmax attention › attention sink