single mention · model architecture
Sink-Augmented Sliding Window Attention
Augments sliding-window attention with sinks to enable information exchange across window boundaries over successive layers.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See sliding window attention for the methods filed beside it.
Also called sink-augmented SWA.
Filed under model architecture › token mixer › softmax attention › sliding window attention