Techniquesmodel architecturepositional encoding

specific method · filed under model architecture

2D rotary position embedding

A rotary positional embedding applied to 2D positions, including an axial variant in the supplied evidence.

Also called 2D rotary position embeddings (2D-RoPE), 2D-RoPE, 2D rotary positional embeddings, axial 2D-RoPE with non-causal attention.

sources: 3
models: 2
labs adopt it: 2
strongest: used

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 3

Documented in

DeepSeek-V4.1-Flash Gemma 4 31B

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Rotary Position Embedding for Vision Transformer (Heo et al., 2024) paper arxiv.org
axial 2D-RoPE for vision transformers

Evidence

3 spans quoted from the sources, strongest treatment first.

incorporate both axial 2D-RoPE [Heo et al., 2024] with non-causal attention and 2D absolute positional embeddings

usedmodel architecturein Gemma 4Google DeepMind

Gemma 4 Technical Report·Gemma Team, Google DeepMind·2026-07-02

we replace standard absolute positional embeddings with 2D-RoPE.

usedmodel architecturein DeepSeek-V4.1-FlashDeepSeek

DeepSeek V41 Tech Report·DeepSeek

A vision encoder (DeepSeek-ViT, trained from scratch with 2D-RoPE and 3×3 pixel-unshuffle downsampling)

usedmodel architecturein DeepSeek-ViTDeepSeek

deepseek-ai/DeepSeek-V4.1-Flash · Hugging Face·DeepSeek·2026-09-10

Filed alongside

Other methods under model architecture :: positional encoding.

YaRN Rotary Position Embedding RoPE scaling Learned input-dependent relative position bias No Position Encoding Proportional Rotary Position Embedding Relative attention 2D coordinate-based positional embeddings Gated attention with partial RoPE Omitting RoPE in attention layers Partial RoPE Per-layer-type rotary position scales Temporal-Modality Rotary Position Embedding