specific method · filed under model architecture
Learned input-dependent relative position bias
A learned, input-dependent relative-position term used instead of RoPE; one implementation limits it to the preceding 1,024 tokens.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 2
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
A learned, input-dependent relative-position bias instead of RoPE.
usedmodel architecturein InklingThinking Machines Lab
In the 11 global layers, the released implementation applies the learned bias only over the preceding 1,024 tokens.
usedmodel architecturein InklingThinking Machines Lab
Filed alongside
Other methods under model architecture :: positional encoding.
YaRNRotary Position Embedding2D rotary position embeddingRoPE scalingNo Position EncodingProportional Rotary Position EmbeddingRelative attention2D coordinate-based positional embeddingsGated attention with partial RoPEOmitting RoPE in attention layersPartial RoPEPer-layer-type rotary position scalesTemporal-Modality Rotary Position Embedding