single mention · model architecture
Off-by-one attention
Off-by-one attention is a softmax-attention variant in which each attention head has a learned bias in the denominator of the softmax.1 This bias enables the attention mechanism to pay no attention to any tokens.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Each attention head has a learned bias in the denominator of the softmax, similar to off-by-one attention
— gpt-oss-120b & gpt-oss-20b Model Cardwhich enables the attention mechanism to pay no attention to any tokens
— gpt-oss-120b & gpt-oss-20b Model Card
Filed under model architecture › token mixer › softmax attention