specific method · filed under model architecture
Global attention
Global attention is an attention mechanism in which each token can attend over the entire sequence rather than only a local window.1 It is interleaved with sliding window attention layers in hybrid architectures, motivating hybrids that combine local efficiency with full-sequence access.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Sliding window attention [beltagy2020longformer] restricts each token’s attention scope to a local window rather than the entire sequence
— MiMo-V2-Flash Technical ReportThis naturally motivates hybrid attention architectures that interleave sliding window attention with global attention
— MiMo-V2-Flash Technical Report
Also called Global Attention (GA), Global attention layers.