Relative attention

An approach that represents position with a relative term; the supplied evidence includes a learned term added to pre-softmax logits but does not establish that all mentions use that mechanism.

Also called Relative positional embedding.

sources: 2
model: 1
lab adopt it: 1
strongest: core

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 1core 1

Documented in

Inkling

Evidence

2 spans quoted from the sources, strongest treatment first.

Inkling replaces RoPE with relative attention: a learned relative-position term added to the pre-softmax logits.

coremodel architecturein InklingThinking Machines Lab

thinkingmachines/Inkling — 975B / 41B active · MOE · 1024K ctx·vLLM·2026-07-14

We find that encoding position with a relative positional embedding performs better and extrapolates better to longer sequences than the more widely adopted Rotary Positional Embedding (RoPE).

usedmodel architecturein InklingThinking Machines Lab

Inkling: Our Open-Weights Model·Thinking Machines Lab·2026-07-15

Filed alongside

Other methods under model architecture :: positional encoding.

YaRN Rotary Position Embedding 2D rotary position embedding RoPE scaling Learned input-dependent relative position bias No Position Encoding Proportional Rotary Position Embedding 2D coordinate-based positional embeddings Gated attention with partial RoPE Omitting RoPE in attention layers Partial RoPE Per-layer-type rotary position scales Temporal-Modality Rotary Position Embedding