Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attention

single mention · model architecture

Off-by-one attention

Off-by-one attention is a softmax-attention variant in which each attention head has a learned bias in the denominator of the softmax.1 This bias enables the attention mechanism to pay no attention to any tokens.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Each attention head has a learned bias in the denominator of the softmax, similar to off-by-one attention — gpt-oss-120b & gpt-oss-20b Model Card
  2. which enables the attention mechanism to pay no attention to any tokens — gpt-oss-120b & gpt-oss-20b Model Card

Filed under model architecture › token mixer › softmax attention