single mention · model architecture
Full RoPE in sliding-window attention layers
Sliding-window attention layers retain full rotary positional embeddings rather than an indicated partial form.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See positional encoding for the methods filed beside it.
Also called SWA layers keep full RoPE.
Filed under model architecture › positional encoding