Open Models Almanac
Methodsmodel architecturepositional encoding

specific method · filed under model architecture

Learned relative attention

Relative-position information is represented through a learned term or embedding rather than RoPE; the evidence does not establish that the described implementations are identical.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See positional encoding for the methods filed beside it.

Also called relative attention, Relative positional embedding.

sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Thinking Machines Lab

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 1

Inkling

Thinking Machines Labadopted2 sources

We find that encoding position with a relative positional embedding performs better and extrapolates better to longer sequences than the more widely adopted Rotary Positional Embedding (RoPE).

usedmodel architecturein InklingThinking Machines Lab
1 more from this model

Inkling replaces RoPE with relative attention: a learned relative-position term added to the pre-softmax logits.

coremodel architecturein InklingThinking Machines Lab
↑