Inkling
Thinking Machines LabWe find that encoding position with a relative positional embedding performs better and extrapolates better to longer sequences than the more widely adopted Rotary Positional Embedding (RoPE).
1 more from this model
Inkling replaces RoPE with relative attention: a learned relative-position term added to the pre-softmax logits.