Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionattention sink

specific method · filed under model architecture

Learnable attention sink token

A learnable attention sink token gives each attention head an extra learnable sink logit to represent the attention-sink behavior that otherwise falls on the first token.1 The sink logit competes with normal key positions in the attention softmax, absorbing attention mass instead of the first real token acting as an implicit sink.2

Trade-off It adds parameters and implementation complexity and, in the reported test, did not yield a clear or consistent improvement, so it was not adopted.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. each attention head is given an additional learnable sink logit, which competes with normal key positions in the attention softmax — MiniMax Sparse Attention
  2. which competes with normal key positions in the attention softmax — MiniMax Sparse Attention
  3. Given its additional parameters, implementation complexity, and the fact that it does not fully suppress first-token sink behavior, we do not include the learnable attention sink in the final recipe — MiniMax Sparse Attention

Also called learnable attention sink, a learnable attention sink token.

sources
2
models
2
lab adopt it
1
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 1not used 1

MiniMax-M3

MiniMaxnot used1 source

we do not include the learnable attention sink in the final recipe

not usedmodel architecturein MiniMax Sparse AttentionMiniMax
↑