Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionsliding window attention

single mention · model architecture

Sink-Augmented Sliding Window Attention

Augments sliding-window attention with sinks to enable information exchange across window boundaries over successive layers.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See sliding window attention for the methods filed beside it.

Also called sink-augmented SWA.

Filed under model architecture › token mixer › softmax attention › sliding window attention