Open Models Almanac
Methodsmodel architecturepositional encoding

single mention · model architecture

Full RoPE in sliding-window attention layers

Sliding-window attention layers retain full rotary positional embeddings rather than an indicated partial form.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See positional encoding for the methods filed beside it.

Also called SWA layers keep full RoPE.

Filed under model architecture › positional encoding